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FOREWORD 

a.  This  technical  report,  BDM/ABQ-86-0090-TR,  is  submitted  by  The 
BOM  Corporation,  1801  Randolph  Road  SE,  Albuquerque,  New  Mexico  87106 
to  the  Air  Force  Operational  Test  and  Evaluation  Center,  Kirtland  Air 
Force  Base,  Albuquerque,  New  Mexico  87117-7001.  This  submission  Is 
in  compliance  with  the  requirements  of  paragraph  7.2  of  Subtask 
Statement  412/1,  titled  "Software  Supportability  Risk  Assessment: 
Pilot  Application." 

b.  This  report  is  the  result  of  effort  by  Dr.  David  Peercy 
(Technical  Leader)  and  Mr.  Walter  Hui^r,  Jr.  (Task  Leader),  of  The 
BOM  Corporation.  The  primary  Subtisk  Statement  Project  Officer  is 
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SECTION  I 
INTRODUCTION 


1.1  PURPOSE  OF  GUIDELINES. 

Tht  purposes  of  the  guidelines  presented  In  this  report  are 
to: 

(1)  Document  the  procedures  necessary  to  adapt  the  current 
Air' Force  Operational  Test  and  Evaluation  Center  (AFOTEC) 
software  support  ability  test  methodologies  to  the  con¬ 
cepts  of  risk  assessment  (see  section  1.4  for  appropriate 
references) 

(2)  Document  the  procedures  required  to  perform  a  Software 

Life  Cycle  Process  (SCLP)  supportablllty  evaluation, 
thereby  establishing  a  third  tool  available  to  AFOTEC  for 
use  in  evaluating  software  supportablllty  (see 

figure  l-l). 

1.2  OVERVIEW  OF  RISK  ASSESSMENT. 

a.  AFOTEC  has  the  responsibility  for  conducting  operational  test 
and  evaluation  (OTtE)  of  assets  entering  the  Air  Force  Inventory. 
AFOTEC  has  developed  and  Implemented  various  software  OTIE  method¬ 
ologies.  Two  of  these  methods.  Software  Product  maintainability 
evaluation  and  Software  Support  Resources  evaluation,  are  shown  In 
figure  1-1  to  Illustrate  how  they  relate  to  the  overall  evaluation  of 
software  supportablllty.  Over  the  past  several  years,  these  two 
methods  have  matured  and  have  bKome  the  Air  Force  standard  for 
evaluating,  software  supportablllty.  Each  of  these  developed  methods 
evaluates  specific  characteristics  of  the  supportablllty  aspects  of 
delivered  software  products  and  software  support  resources.  These 
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Figure  1-1.  Elements  of  Software  Supportability  Evaluations 
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Stand-alone  evaluations  provide  AFOTEC  with  Information  to  Identify 
particular  software  supportablllty  deficiencies,  but  do  not  Identify 
overall  risk  associated  with  contractor  or  military  ownership  and 
organic  maintenance  of  contractor-delivered  software. 

b.  AFOTEC s  concern  about  the  need  of  a  risk  assessment  method 
which  provides  software  testers  with  areas  which  require  testing 
emphasis,  and  decision  makers  with  an  assessment  of  the  software  sup- 
portability  risk,  has  resulted  In  the  development  of  the  Risk  Assess¬ 
ment  Methodology  for  Software  Supportablllty  (RAMSS).  This 
methodology  uses  the  Software  Product  and  Software  Support  Resources 
evaluations  eentloned  above.  In  addition  to  a  third  method  (docu¬ 
mented  in  this  report)  called  a  Software  Life  Cycle  Process  (SCLP) 
evaluation,  to  produce  an  overall  software  supportablllty  risk.  The 
following  paragraphs  discuss  the  evolution  and  general  application  of 
the  risk  assessment  process. 

1.2.1  Concept  Development. 

a.  Since  1982,  AFOTEC  has  been  analyzing  the  problem  of  how  to 
assess  the  risk  to  the  Air  Force  of  supporting  software  acquired  for 
weapon  systems.  A  concept  for  computer  resources  risk  assessment 
during  operational  test  and  evaluation  was  proposed  In  1983 
(reference  1.4.16).  Several  Issues  evolved  from  this  proposal. 
First,  the  assessed  risk  should  reflect  software  supportablllty 
Impact  upon  the  system  at  a  level  appropriate  for  AFOTEC  reporting 
requirements.  Second,  supportablllty  is  a  concern  for  both  the  user 
and  the  supporter.  Any  defined  risk  of  software  supportablllty 
should  reflect  some  aspect  of  user  risk  and  supporter  risk.  Third, 
current  AFOTEC  methods  of  evaluating  software  supportablllty  should 
be  Integrated  Into  the  risk  assessment  method.  Also,  the  risk 
assessment  method  should  be  adaptable  to  Include  other  AFOTEC 
concerns  such  as  software  maturity  and  software  reliability. 
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b.  This  initial  concept  proposal  provided  AFOTEC  with  justifi¬ 
cation  to  study  the  feasibility  of  developing  and  Implementing  a  risk 
assessment  methodology  for  software  supportablllty  (RAMSS).  The 
approach  for  this  study  (references  1.4.2,  1.4.3,  1.4.4)  Included: 

(1)  Literature  review  and  assemblage  of  a  data  base  of 
relevant  tools,  techniques  and  methods 

(2)  Analysis  of  relevant  tools,  techniques,  and  methods  for 
feasibility  of  application  to  AFOTEC's  needs 

(3)  Development  of 'a  framework  for  assessing  software  sup- 
portability  risk  along  with  a  preliminary  set  of  risk 
measures. 

c.  The  primary  conclusion  from  this  feasibility  study  was  that  a 
RAMSS  could  be  developed  based  upon  the  framework  derived  as  part  of 
the  study.  However,  there  were  still  several  technical  Issues  which 
needed  to  be  resolved.  Of  these  Issues,  the  major  one  concerned  the 
need  to  establish  a  baseline  against  which  to  measure  risk.  Since 
risk  was  defined  (for  this  study)  as  "the  potential  for  realization 
of  unwanted,  negative  consequences  of  an  event,”  It  was  necessary  to 
have  a  baseline  of  software  support  activities  In  order  to  tell  when 
a  consequence  may  be  negative.  This  baseline,,  called  an  historical 
maintenance  profile,  reflects  how  software  support  resources  are 
being  used  to  perform  the  software  support  activities.  Given  this 
Information,  the  framework  recommended  by  the  feasibility  study  could 
be  Used  to  compute  measures  of  risk  and  incorporate  the  issues 
proposed  in  1983. 
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1.2.2  Methodology  Requirements  (Inputs).  Figure  1-2  illustrates 
interfaces  with  the  RAMSS.  The  inputs  consist  of: 

(1)  The  historical  profile  of  software  maintenance  activity 

(2)  A  user/supporter  estimate  on  planned  software  maintenance 
changes  and  support  resource  requirements  for  the  soft¬ 
ware  system  being  evaluated 

(3)  An  evaluation  of  software  support  capabilities  using 

current  AFOTEC  methods. 

1.2.3  Methodology  Analysis.  The  RAMSS  inputs  are  combined  and 
analyzed,  and  measures  of  risk  computed  for  the  system  being 
evaluated. 

1.2.4  Methodology  Benefits  (Results). 

a.  The  major  results  of  the  RAMSS  are  also  Illustrated  in 

figure  1-1.  These  results  include: 

(1)  The  software  supportablllty  risk  measure  which  quantifies 
the  probability  of  the  user/supporter  baseline  estimate 
not  being  accomplished  with  current  software  support 
capabilities 

(2)  The  capability  to  identify  the  impact  of  the  software 

supportablllty  risk  as  high,  medium,  or  low 

(3)  The  identification  of  the  drivers  of  the  software  sup- 

portability  risk 

(4)  The  projection  of  alternative  choices  for  risk  reduction 
(for  Instance,  by  improving  certain  aspects  of  current  or 
projected  software  support  capabilities). 
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b.  With  this  information,  the  decision  maker  can  assess  the 
effect  of  software  supportabillty  upon  system  suitability  and 
effectiveness.  In  addition,  detailed  data  are  available  to  help 
answer  specific  questions  such  as  why  particular  areas  of  software 
supportabillty  are  drivers  and  how  the  measured  risk  can  be  reduced 
to  an  acceptable  level. 


1.2.5  Baseline  Definition  and  ApDlication. 


a.  As  discussed  above,  a  key  element  to  the  risk  assessment 
process  Is  recognition  that  software  supportabillty  is  important  both 
to  the  user  and  to  the  supporter  of  the  software.  Therefore  any  risk 
assessment  methodology  which  Ignores  the  Interests  of  one  of  these 
parties  may  estimate  a  risk  that  Is  unacceptable  to  the  other.  In  an 
attempt  to  bridge  this  gap,  the  RAMSS  input  requires  a  User/Supporter 
Baseline  Estimate  be  established,  and  that  the  evaluations  of  soft¬ 
ware  support  capabilities  be  made  against  that  Baseline. 


b.  The  User /Supporter  Baseline  Estimate  uses  Inputs  from  both  the 
user  (using  command)  and  the  supporter  (supporting  command).  The 
estimate  Includes  an  understanding  of  the  software  block  release 
cycle,  projected  software  support  personnel  (numbers  and  types),  and 
anticipated  software  change  request  activity  for  each  block  release. 
Details  of  the  User /Supporter  Baseline  Estimate  are  contained  In 
section  VI  of  this  report. 


c.  The  current  AFOTEC  methods  for  evaluating  software  support- 
ability  do  not  consider  in  a  direct  manner- the  effect  of  an  estimated 
baseline.  The  establishment  of  an  estimated  baseline  is  critical  to 
risk  assessment  because  It  (1)  provides  a  means  to  Judge  how  well  the 
measured  risk  agrees  with  the  estimated  risk  (which  Is,  In  some 
sense,  "acceptable"  to  both  the  user  and  the  supporter),  and 
(2)  quantifies  the  options  required  to  lower  the  measured  and 
acceptable  risks  (a  desired  result  of  the  risk  assessment  process). 
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This  report  documents  what  steps  should  be  taken  during  the  evalua¬ 
tion  of  a  system's  software  supportability  to  ensure  that  the  User/ 
Supporter  Baseline  Estimate  is  accounted  for  in  the  risk  assessment 
process. 

1.3  GENERAL  ORGANIZATION  OF  GUIDELINES. 

The  remainder  of  this  report  is  organized  into  five  additional 
sections,  plus  an  appendix  that  contains  materials  necessary  to  use 
the  software  life  cycle  process  method  (tool)  with  the  RAMSS.  Report 
sections  satisfy  the  following  objectives: 

(1)  Section  II  gives  an  overview  of  the  adaptation  guidelines 
recommended  by  this  report 

(2)  Section  III  contains  a  discussion  of  the  adaptation  of 
the  Software  Product  evaluation  method 

• 

(3)  Section  IV  contains  a  discussion  of  the  adaptation  of  the 
Software  Support  Resources  evaluation  method 

(4)  Section  V  Introduces  and  describes  a  method  to  evaluate 
the  Software  Life  Cycle  Process  for  characteristics  of 
software  supportab  1 1 1  ty 

(5)  Section  VI  provides  details  on  obtaining  a  User/Supporter 
Baseline  Estimate 

(6)  Appendix  A  is  an  Evaluator's  Guide  to  the  performance  of 
the  Software  Life  Cycle  Process  (SCLP)  evaluation  method. 
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1.5  TERMS  AND  ABBREVIATIONS. 


AF  Air  Force 

AFB  Air  Force  Base 

AFOTEC  Air  Force  Operational  Test  and  Evaluation  Center 
AISF  Avionics  Integration  Support  Facility 

ALC  Air  Logistics  Center 

ASSET  AFOTEC  Software  Support  Evaluation  Tool 

BMOP  BMOP  Statistical  Software  (NOTE:  BMDP  Is  a  name,  not  an 
acronym. ) 

C«E  CommunlcatlonS'Electronlcs 

COR  Critical  Design  Review 

CRISP  Computer  Resources  Integrated  Support  Plan 

CRLCMP  Computer  Resources  Life  Cycle  Management  Plan 

CRWG  Computer  Resources  Working  Group 

OoO  Department  of  Defense 

DSE  Deputy  for  Software  Evaluation 

DTIE  Development  Test  and  Evaluation 

ECS  Embedded  Computer  System 

FCA  Functional  Configuration  Audit 

HQ-TAC  Headquarters  Tactical  Air  Command 

IVliV  Independent  Verification  and  Validation 

JTIDS  Joint  Tactical  Information  Distribution  System 

0/S  CMP  Operational /Support  Configuration  Management  Procedures 

OT&E  OperatiQTial  Test  and  Evaluation 

PCA  Physical  Configuration  Audit 

PDR  Preliminary  Design  Review 

PMD  Program  Management  Directive 

PMP  Program  Management  Plan 

PMPC  Person  Months  Per  Change 

PMRT  Program  Management  Responsibility  Transfer 

RA  Risk  Assessment 


RAMSS  Risk  Assessment  Methodology  for  Software  Supportablllty 

RFP  Request  for  Proposal 
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SLCP 

SON 

S/W 

SS 

SS 

SSR 

SSR 

STM 

TEMP 

TRR 

UR-ALC 


Software  Life  Cycle  Process 
Statement  of  Need 
Software 

Software  Supportablllty 

Software  Support 

Software  Specification  Review 

Software  Support  Resources 

Software  Test  Manager 

Test  and  Evaluation  Master  Plan 

Test  Readiness  Review 

Warner  Robins  Air  Logistics  Center 
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SECTION  II 

OVERVIEW  OF  ADAPTATION  GUIDELINES 


2.1  INTRODUCTION. 

a.  The  guidelines  for  adapting  the  AFOTEC  Software  Supportablllty 
Evaluation  to  the  requirements  of  the  RAMSS  focus  on  two  primary 
aspects.  First,  the  current  evaluations  of  software  product  and  soft¬ 
ware  support  resources  characteristics  should  be  appropriately 
updated.  In  particular,  the  User/Supporter  Baseline  Estimate  should 
be  Integrated  Into  the  evaluation  procedure.  Second,  the  Software 
Life  Cycle  Process  (SLCP)  evaluation  should  be  conducted  to  provide 
proper  evaluation  depth  In  the  areas  of  software  project  management 
and  software  configuration  managenwnt.  The  software  supportablllty 
evaluation  hierarchy  Is  shown  In  figure  1-1  of  section  I. 

b.  The  current  AFOTEC  procedures  for  software  supportablllty 
evaluations  (reference  1.4.8)  were  reviewed.  A  summary  of  the 
resulting  adaptation  guidelines  Is  presented  In  sections  2.2  and  2.3. 
A  summary  of  the  recommended  guidelines  for  adding  the  SLCP  evalua¬ 
tion  to  AFOTEC s  evaluation  procedures  Is  presented  In  section  2.4. 
A  summary  of  the  recommended  guidelines  for  derivation  and  Integra¬ 
tion  of  the  User/Supporter  Baseline  Estimate  Is  presented  In 
section  2.5.  Appropriate  details  of  these  guidelines  are  presented 
In  sctlons  III,  IV,  V,  VI  and  appendix  A.  The  major  guidelines  are 
summarized  In  figure  2-1. 

2.2  SOFTWARE  PRODUCT  EVALUATION. 

a.  A  review  of  current  AFOTEC  evaluation  for  software  product 
supportablllty  characteristics  (see  Volume  III  of  reference  1.4.8) 
Indicated  there  were  no  significant  changes  necessary  to  Integrate 
with  the  RAMSS.  The  User/Supporter  Baseline  Estimate  should 
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niinimany  affect  the  conduct  of  or  evaluation  results  for  the  soft¬ 
ware  product  evaluation.  The  current  guidelines  for  conduct  of  the 
evaluation  appear  to  be  adequate*  except  that  the  User/Supporter 
Baseline  Estimate  should  be  discussed  during  the  evaluator  calibra¬ 
tion  step. 
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Figure  2-1.  Summary  of  Adaptation  Guidelines 

b.  One  minor  additional  step  is  required  in  the  software  product 
evaluation  procedure.  When  the  evaluation  is  complete*  the  evalua¬ 
tion  results  required  by  the  RAMSS  must  be  compu^’ed  and  entered  into 
the  RAMSS  data  base  through  procedures  described  in  the  RAMSS  User's 
Handbook  (reference  1.4.6).  The  current  automated  tools  which 
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support  this  evaluation  provide  the  necessary  computation  and  report 
capability.  The  evaluation  results  required  Include: 

(1)  Software  Documentation 

a)  Modularity  Rating 

b)  Oescriptiveness  Rating 

c)  Consistency  Rating 

d)  Simplicity  Rating 

e)  Expandability  Rating 

f)  Instrumentation  Rating 

(2)  Software  Source  Code 

a)  Modularity  Rating 

b)  Descriptiveness  Rating 

c)  Consistency  Rating 

d)  Simplicity  Rating 

e)  Expandability  Rating 

f)  Instrumentation  Rating 

Evaluation  scores  are  In  the  real  range  from  1.0  to  6.0. 

2.3  SOFTWARE  SUPPORT  RESOURCES  EVALUATION. 

A  review  of  the  current  AFOTEC  evaluation  for  Software  Support 
Resources  (SSR)  supportablllty  characteristics  resulted  In  the 
following  recommended  guidelines: 

(1)  It  appears  that  the  current  automated  tools  (ASSET)  are 
not  being  as  effectively  utilized  as  possible.  The  RAMSS 
depends  upon  the  capability  of  AFOTEC  to  obtain  evalua¬ 
tion  ratings  at  the  appropriate  hierarchy  level.  This 
level  Is  reflected  In  the  hierarchy  as  supported  by  the 
ASSET  evaluation  tools.  However,  since  RAMSS  only 
depends  upon  the  availability  of  the  corresponding 
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evaluation  ratings,  not  how  they  were  obtained  (e.g., 
from  lower  level,  ASSET-based  questionnaires)  the  use  of 
ASSET  Is  not  directly  required  to  obtain  a  valid  risk 
assessment  using  the  RAMSS.  However,  AFOTEC  personnel 
need  a  more  effective  use  of  the  concepts  and  contents  of 
Volume  V  (see  reference  1.4.8)  or  a  more  effective 
reorientation  to  Volume  V  In  order  to  properly  utilize 
the  ASSET  capabilities. 

(2)  The  User/Supporter  Baseline  Estimate  Is  useful,  and 
should  be  Incorporated  Into  the  SSR  evaluation  during  the 
early  evaluation  orientation  and  discussion.  The  main 
use  of  the  estimate  appears  to  be  In  facilitating  neces¬ 
sary  discussion  on  resource  (personnel,  systems, 
facilities)  requirements.  This  results  In  better  plan¬ 
ning  for  actual  resource  requiretnents  which  become  part 
of  the  SSR  evaluation  discussion. 

(3)  Another  minor  additional  step  Is  required  In  the  SSR 

evaluation  procedure.  When  the  evaluation  Is  complete, 

the  evaluation  results  required  by  the  RAMSS  must  be  com¬ 
puted  and  entered  Into  the  RAMSS  data  base  through 
procedures  described  In  the  RAMSS  User's  Handbook 

(reference  1.4.6).  The  current  ASSET  automated  tools 
which  support  this  evaluation  provide  adequate  computa¬ 
tion  and  report  capability.  The  evaluation  results 
required  Include: 


a)  Personnel/Resources 
Management  Rating 
Technical  Rating 
Support/Clerical  Rating 
Contractor  Rating 


THE  BDM  CORPORATION 


BDM/ABQ-86-0090-TR 


b)  Support  Systems  Resources 
Host  Processors  Rating 
Software  Bench (es)  Rating 

Laboratory- Integrated  Test  Facilities  Rating 
Operational-Integrated  Test  Facilities  Rating 
Configuration  Management  System  Rating 
Other  Support  Systems  Rating 

c)  Physical  Facilities  Resources 

General  Office/Storage  Facilities  Rating 
Support  Systems  Facilities  Rating 

Evaluation  scores  are  In  the  real  range  from  1.0  to  6.0. 

2.4  SOFTWARE  LIFE  CYCLE  PROCESS  EVALUATION. 

a.  The  Software  Life  Cycle  Process  (SLCP)  evaluation  Is  recom¬ 
mended  In  references  1.4.4  and  1.4.$  to  supplemtnt  the  current  AFOTEC 
software  supportablllty  evaluation.  The  SLCP  evaluation  focuses  upon 
the  project  management  and  configuration  management  •  processes  across 
the  software  life  cycle.  It  Is  recommended  that  AFOTEC  adopt  the 
SLCP  evaluation  procedures  as  described  In  section  V  and  appendix  A. 
The  format  of  appendix  A  was  chosen  to  facilitate  adoption  of  the 
SLCP  as  an  AFOTEC  800-2  pamphlet. 

b.  In  the  process  of  obtaining  Information  for  the  SLCP  evalua¬ 
tion,  the  Software  Test  Manager  {STM)/Deputy  for  Software  Evaluation 
(OSE)  should  establish  one  or  more  versions  of  the  User/Supporter 
Baseline  Estimate.  The  SLCP  evaluation  can  utilize  the  User/Sup¬ 
porter  Baseline  Estimate  as  an  aid  to  assessing  whether  the  support 
activity  can  actually  accomplish  proper  project  and  configuration 
management  In  light  of  the  procurement  and  contractor  project  and 
configuration  management  practices.  Project  management  charac¬ 
teristics  of  evaluation  include  planning,  organization  structure. 
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design  methods,  Implementation  methods,  test  strategies  and  organiza¬ 
tion  Interfaces.  Configuration  management  characteristics  of 
evaluation  Include  configuration  identification,  configuration  con¬ 
trol,  status  accounting,  and  audit  review.  Each  of  the  charac¬ 
teristics  is  evaluated  relative  to  the  life  cycle  activities  of 
procurement,  con  tractor /development,  and  operational /support. 

c.  It  is  recommended  that  the  AFOTEC  STM  and/or  OSE  be 
responsible  for  the  conduct  of  the  SLCP  evaluation.  Information  to 
help  the  overall  assessment  can  be  gathered  from  many  sources  over 
the  life  of  OT&E  involvement  with  a  system  evaluation.  The  Test  and 
Evaluation  Master  Plan  (TEMP),  Computer  Resources  Integrated  Support 
Plan  (CRISP),  Operational /Support  Configuration  Management  Procedures 
(0/S  CMP),  the  Computer  Resources  working  Group  (CRWG),  and  the  many 
Interface  meetings  and  program  iivlews  attended  by  STM/DSE  or  repre¬ 
sentatives  are  useful  sources  of  life  cycle  information.  The  SLCP 
evaluation  could  be  completed  by  a  set  of  evaluators  designated  by 
the  STM/DSE,  or  by  the  STM/OSE  with  the  help  of  the  collected  data 
and  other  expert  personnel.  The  information  collected  by  the  STM/DSE 
should  be  entered  into  a  STM/OSE  software  OT&E  notebook  containing 
the  SLCP  questions  for  frequent  reference  and  use  during  life  cycle 
planning  meetings  involving  OT&E  software  personnel  (e.g.,  the 
STM/OSE). 

d.  The  results  of  the  SLCP  evaluation  should  be  entered  into  the 
RAMSS  evaluation  data  base  at  least  for  the  characteristics  required 
by  the  RAMSS,  and  at  the  lower  levels  if  so  desired.  The  minimum 
data  required  includes: 

(1)  Project  Management 

a)  Planning  Rating 

b)  Organizational  Structure  Rating 

c)  Design  Methods  Rating 

d)  Implementation/Coding  Methods  Rating 
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e)  Test  Strategies  Rating 

f)  Project  Interfaces  Rating 

(2)  Configuration  Management  ■ 

a)  Identification  Rating 

b)  Control  Rating 

c)  Status  Accounting  Rating 

d)  Audit/Review  Rating 

Evaluation  scores  are  In  the  real  range  from  1.0  to  6.0. 

2.5  USER/SUPPORTER  BASELINE  ESTIMATE. 

a.  The  User/Supporter  Baseline  Estimate  is  derived  from  inter¬ 
action  among  AFOTEC,  Using  Command,  and  Supporting  Connand  personnel. 
The  RAMSS  automated  support  tools  can  be  used  to  assist  In  deriving  a 
draft  Estimate  from  historical  data.  The  Estimate  should  be  obtain¬ 
able  through  normal  program  OTftE  planning  and  evaluation  functions, 
perhaps  augmented  by  one  or  two  short  site  visits  and  telephone 
contacts.  The  resulting  Estimate  consists  of  the  following  Informa¬ 
tion: 

(1)  System  Identification  Data:  names  of  the  system,  soft¬ 
ware,  using  and  supporting  commands 

(2)  Software  Support  Concept:  block  release  duration  and 
overlap,  and  personnel  requirements 

(3)  Baseline  Change  Profile:  change  request  totals  for  up  to 
three  block  releases  by  type,  complexity  and  priority. 

b.  The  User /Supporter  Baseline  Estimate  may  not  be  extensively 
used  in  the  Software  Product  evaluation,  the  Software  Support 
Resources  evaluation  and  the  Software  Life  Cycle  Process  evaluation. 
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SECTION  III 

SOFTWARE  RROOUCT  EVALUATION  GUIDELINES 
3.1  INTRODUCTION. 

a.  The  software  product  evaluation  has  a  top  level  hierarchy  as 
shown  In  figure  3-1  (volume  III,  reference  1.4.8).  This  top  level 
hierarchy  corresponds  to  the  level  of  evaluation  required  by  the 
RAMSS  (reference  1.4.5).  In  particular,  evaluation  results  at 
level  3  of  the  hierarchy  are  required  for  Input  to  the  RAMSS  evalua¬ 
tion  data  base.  The  RAMSS  does  not  require  any  other  specific 
assumptions  of  the  software  product  evaluation. 

b.  Of  the  evaluation  methodologies  used  by  AFOTEC,  the  RAMSS 
probably  has  the  least  effect  on  the  software  product  evaluation. 
After  examining  the  questions  used  during  the  software  product 
evaluation.  It  seemed  that  the  level  of  responses  were  In  general  too 
detailed  to  be  Influenced  by  the  User /Supporter  Baseline  Estimate. 
For  example,  many  questions  require  forced  responses  based  upon  the 
software  development  techniques  used  (I.e.,  language  type,  existence 
of  preface  block  Information,  top  down  structured  programming)  which 
are  Independent  of  the  User /Supporter  Baseline  Estimate  of  software 
support  requirements.  The  responses  to  some  questions  In  the  Instru¬ 
mentation  category  may  be  Influenced  by  the  evaluator's  knowledge  of 
a  high  or  low  risk  from  the  User/Supporter  Baseline  Estimate; 
however.  It  Is  not  possible  to  make  a  direct  correlation  between  the 
questions  and  the  estimate.  The  primary  value  to  understanding  the 
User/Supporter  Baseline  Estimate  In  the  software  product  evaluation 
process  Is  the  additional  system  knowledge  provided  to  the  evaluators 
that  may  point  to  specific  problems  and/or  areas  which  require 
special  attention. 
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Figure  3-1.  Software  Product  Maintainability  Evaluation 
Heirarchy 
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3.2  REQUIREMENTS. 

The  adaptation  of  the  software  product  evaluation  to  accommo* 
date  the  RAMSS  requirements  Is  subject  to  several  constraints  and/or 
requirements  listed  below. 

(1)  Reoulrement  1;  The  software  product  hierarchy  of  evalua¬ 
tion  criteria  must  agree  with  the  hierarchy  as  presented 
In  figure  3-1  through  level  3.  Each  characteristic 
(level  3)  shall  be  measured  on  the  standard  AFOTEC  scale 
which  allows  for  real  valued  responses  from  1.0  to  6.0. 

(2)  Requirement  2;  The  current  procedures  for  evaluating  the 
software  product  characteristics  must  not  be  signifi¬ 
cantly  Impacted  by  the  RAMSS  requirements.  The  only 
modifications  to  the  current  evaluation  procedures 
Include  the  use  of  the  User/Supporter  Baseline  Estimate 
as  an  estimate  of  the  post  deployment  software  support 
block  release  schedule,  personnel  requirements,  and  level 
of  change  activity. 

(3)  Requirement  3;  Evaluation  results  must  be  entered  Into 
the  RAMSS  evaluation  data  base  using  the  automated 
support  tools  described  In  the  RAMSS  User's  Handbook 
(reference  1.4.6).  The  evaluation  results  correspond  to 
the  level  3  characteristics  of  figure  3-1. 

3.3  GUIDELINES. 

The  guidelines  for  adapting  the  software  product  evaluation 
for  the  RAMSS  are  organized  Into  the  following  subsections:  Software 
Product  Evaluation  Procedure,  Software  Product  Evaluation  Materials, 
and  Software  Product  and  RAMSS  Interface. 

)' 
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3.3.1  Software  Product  Evaluation  Procedures.  The  Evaluation 
Procedure  consists  of  the  following  four  distinct  phases: 

(1)  Phase  I  -  plan  for  the  evaluation 

(2)  Phase  II  -  calibrate  the  evaluators 

(3)  Phase  III  •  evaluate  the  software  product  maintainability 

(4)  Phase  IV-  analyze  and  assess  the  results. 

Each  of  the  phases  Is  modified  to  Include  RAMSS  considerations  in  the 
following  way: 

(1)  Phase  I  •  STM/OSE  assure  that  a  credible  User /Supporter 
Baseline  Estimate  Is  available.  If  not,  plan  to  obtain  a 
draft  version  based  upon  historical  data  using  the  RAMSS 
automatic  support  tools.  Iterate  the  draft  version  among 
Using  Command  and  Supporting  Command  personnel  until 
there  Is  a  reasonable  agreement  of  the  estimated  software 
support  baseline.  See  section  VI  for  more  details  on  the 
baseline. 

(2)  Phase  II  -  the  User/Supporter  Baseline  Estimate  Is 
explained  to  the  evaluators  during  the  calibration  brief¬ 
ing  for  the  software  product  evaluation.  Actual  Estimate 
data  probably  has  only  a  minor  direct  effect  of  providing 
additional  system  knowledge  during  the  calibration  and 
eventual  evaluation.  Most  of  the  value  Is  from  communi¬ 
cation  and  discussion  resulting  from  derivation  of  the 
Estimate. 
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(3)  Phase  III  -  no  specific  RAMSS  Impact  during  this  phase. 

(4)  Phase  IV  -  the  res’^lts  of  the  software  product  evaluation 
for  the  level  3  characteristics  are  entered  Into  the 
RAMSS  evaluation  data  base  for  subsequent  risk  analysis. 

3.3.2  Evaluation  Materials.  The  only  additional  materials  required 
for  the  software  product  evaluation  are  the  User/Supporter  Baseline 
Estimate  and  a  current  software  supportablllty  hierarchy  diagram  (to 
level  3  characteristics). 

3.3.3  Software  Product  and  RAMSS  Interface. 

a.  The  following  twelve  software  product  characteristic  evalua* 
tion  scores.  In  the  real  range  of  1.0  to  6.0,  must  be  entered  Into 
_  the  RAMSS  evaluation  data  base  (reference  1.4.6): 

(1)  Documentation 
Modularity  Rating 
Oescriptiveness  Rating 
Consistency  Rating 
Simplicity  Rating 
Expandability  Rating 
Instrumentation  Rating 

(2)  Source  Code 
Modularity  Rating 
Oescriptiveness  Rating 
Consistency  Rating 
Simplicity  Rating 
Expandability  Rating 
Instrumentation  Rating 
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b.  The  RAMSS  data  base  :>houid  contain  the  User/Supporter  Basel  ine 
Estimate.  A  printed  report  can  be  obtained  for  use  during  the  Soft¬ 
ware  Product  evaluation  (reference  1.4.6). 


IV.  Software  Support  Resources  Evaluation 
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SECTION  IV 

SOFTWARE  SUPPORT  RESOURCES  EVALUATION  GUIDELINES 

4.1  INTRODUCTION. 

a.  The  Software  Support  Resources  (SSR)  evaluation  has  a  top 
level  hierarchy  as  shown  in  figure  4-1  (reference  1.4.8,  volume  V). 
This  top  level  hierarchy  corresponds  to  the  level  of  evaluation 
results  required  by  the  RAMSS  (reference  1.4.5).  In  particular, 
evaluation  results  at  the  level  3  of  the  hierarchy  are  required  for 
input  to  the  RAMSS  evaluation  data  base.  The  RAMSS  does  not  require 
any  other  specific  assumptions  of  the  SSR  evaluation. 

b.  Adequate  planning  for  software  support  resources  depends  upon 
the  expected  maintenance  change  activity.  If  many  changes  which  tend 
to  be  complex  enhancements  are  expected,  then  the  resource  require¬ 
ments  might  be  extensive.  If  few  changes  of  low  complexity  are 
expected,  then  resource  requirements  might  be  minimal.  The  required 
schedule  for  releasing  changes  to  the  field  also  affects  the  software 
support  resource  requirements.  These  concepts  are  part  of  the  User/ 
Supporter  Baseline  Estimate.  This  Estimate  is  used  during  the  soft¬ 
ware  supportability  evaluation  orientation/calibration  of  the 
evaluators. 

4.2  REQUIREMENTS. 

The  adaptation  of  the  SSR  evaluation  to  accommodate  the  RAMSS 
requirements  is  subject  to  several  constraints  and/or  requirements 
listed  below. 

(1)  Requirement  1;  The  SSR  hierarchy  of  evaluation  criteria 
must  agree  with  the  hierarchy  as  presented  in  figure  4-1 
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through  level  3.  Each  characteristic  (level  3)  shall  be 
measured  on  the  standard  AFOTEC  scale  which  allows  for 
real  valued  scores  from  1.0  to  6.0. 

(2)  Requirement  2;  The  current  procedures  for  evaluating  the 
SSR  characteristics  must  not  be  significantly  Impacted  by 
the  RAMSS  requirements.  The  only  modifications  to  the 
current  evaluation  procedures  include  the  use  of  the 
User/Supporter  Baseline  Estimate  as  an  estimate  of  the 
post  deployment  software  support  block  release  schedule, 
personnel  requirements  and  level  of  change  activity. 

(3)  Requirement  3;  Evaluation  results  must  be  entered  into 
the  RAMSS  evaluation  data  base  using  the  automated 
support  tools  described  in  the  RAMSS  User's  Handbook 
(reference  1.4.6).  The  evaluation  results  correspond  to 
the  level  3  characteristics  of  figure  4-1. 

4.3  GUIDELINES. 

The  guidelines  for  adapting  the  SSR  evaluation  for  the  RAMSS 
are  organized  into  the  following  subsections:  SSR  Evaluation 

Procedure,  SSR  Evaluation  Materials,  SSR  and  RAMSS  Interface. 

4.3.1  SSR  Evaluation  Procedure.  The  SSR  Evaluation  Procedure 
consists  of  the  following  four  phases. 

(1)  Phase  I  -  plan  for  the  SSR  evaluation 

(2)  Phase  II  -  generate  the  SSR  evaluation  questionnaire 

(3) '  Phase  III  -  evaluate  the  capability  of  the  SSR  to  satisfy 

each  support  requirement 
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(4)  Phase  IV  -  combine  the  evaluation  results  into  an  overall 
assessment. 


Each  of  the  phases  is  modified  to  include  RAMSS  considerations  in  the 
following  way: 

(1)  Phase  I  -  STM/OSE  assure  that  a  credible  User/Supporter 
Baseline  Estimate  is  available.  If  not,  plan  to  obtain  a 
draft  version  based  upon  historical  data  using  the  RAMSS 
automatic  support  tools.  Iterate  the  draft  version 
among  Using  Command  and  Supporting  Command  personnel 
until  there  is  a  reasonable  agreement  of  the  estimated 
software  support  baseline.  See  section  VI  for  more 
details  on  the  baseline. 

(2)  Phase  II  -  no  specific  RAMSS  impact  during  this  phase. 

(3)  Phase  III  -  the  User /Supporter  Baseline  Estimate  is 
explained  to  the  evaluators  prior  to  conduct  of  the  SSR 
evaluation.  Most  of  the  benefit  is  from  communication 
and  discussion  during  derivation  of  the  Estimate.  Actual 
estimate  data  probably  have  only  a  minor  direct  effect 
providing  additional  system  knowledge  during  evaluation. 

(4)  Phase  IV  -  the  results  of  the  SSR  evaluation  for  the 
level  3  characteristics  are  entered  Into  the  RAMSS 
evaluation  data  base  for  subsequent  risk  analysis. 

4.3.2  SSR  Evaluation  Materials.  The  only  additional  materials 
required  for  the  SSR  evaluation  are  the  User/Supporter  Baseline 
Estimate  and  a  current  software  supportability  hierarchy  diagram  (to 
level  3  characteristics). 
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4.3.3  SSR  and  RAMSS  Interface. 

a.  The  following  twelve  SSR  characteristic  evaluation  scores.  In 
the  real  range  of  1.0  to  6.0,  must  be  entered  Into  the  RAMSS  evalua¬ 
tion  data  base  (reference  1.4.6): 

(1)  Personnel  Resources 
Management  Rating 
Technical  Rating 
Support/Clerical  Rating 
Contractor  Rating 

(2)  Support  Systems  Resources 
Host  Processors  Rating 
Software  Bench (es)  Rating 
Laboratory-Integrated  Test  Facilities  Rating 
Operational -Integrated  Test  Facilities  Rating 
Configuration  Management  Systeni(s)  Rating 
Other  Support  System(s)  Rating 

(3)  Physical  Facility  Resources 

General  Office/Storage  Facilities  Rating 
Support  Systems  Physical  Facilities  Rating 

b.  Some  of  the  twelve  characteristics  may  not  be  evaluated  (e.g., 
contractor  personnel,  other  support  systems)  because  they  are  not 
applicable.  In  other  cases.  It  may  be  that  two  or  more  of  the 
support  systems  are  the  same.  In  which  case  only  one  score  would  be 
computed. 

c.  The  RAMSS  data  base  should  contain  the  User/Supporter  Baseline 
Estimate.  A  printed  report  can  be  obtained  for  use  during  the  SSR 
evaluation  (reference  1.4.6). 


V.  Software  Life  Cycle  Process  Evaluation 
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SECTION  V 

SOFTWARE  LIFE  CYCLE  PROCESS  EVALUATION  GUIDELINES 

5.1  INTRODUCTION. 

a.  Since  AFOTEC  does  not  currently  evaluate  Software  Life  Cycle 
Process  (SLCP)  characteristics*  this  part  of  the  software  supporta- 
bility  evaluation  consists  of  new  evaluation  criteria  along  with 
reconinendations  for  the  procedures*  source  materials*  and  responsible 
personnel  necessary  to  accomplish  the  evaluation.  Adapting  and 
integrating  the  new  criteria  and  procedures  within  the  current  AFOTEC 
evaluation  structure  is  required  In  order  to  obtain  the  necessary 
evaluation  scores  for  input  to  the  RAMSS. 

b.  The  complete  set  of  criteria  and  guidelines  for  their  use  are 
presented  in  appendix  A.  This  section  provides  a  concise  summary  of 
the  requirements  for  the  SLCP  evaluation  criteria*  the  structure  of 
the  criteria*  and  guidelines  for  conducting  the  evaluation  of  the 
software  life  cycle  for  these  criteria. 

5.2  REQUIREMENTS. 

The  SLCP  evaluation  criteria  and  the  associated  evaluation 
procedures  are  subject  to  several  constraints  and/or  requirements 
listed  below. 

(1)  Requirement  1;  The  SLCP  evaluation  criteria  shall  be 
presented  in  the  form  of  a  hierarchy  of  characteristics 
such  that  the  evaluation  Is  conducted  at  the  lowest  level 
with  results  accumulating  to  higher  levels  of  the  hier¬ 
archy.  Each  lowest  level  characteristic  shall  be. 
measured  on  the  standard  AFOTEC  scale  of  1  to  6.  The 
SLCP  hierarchy  (through  level  3)  shall  conform  to  the 
criteria  shown  in  figure  S^-l. 
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Figure  5-1.  Software  Life  Cycle  Process  Evaluation  Hierarchy 
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(2)  Requirement  2;  The  procedures  for  evaluating  the  SLCP 

criteria  shall  integrate  with  the  general  AFOTEC  0T4E 

process  and  specifically  with  the  current  software  sup- 
portability  evaluation  process.  The  source  materials, 
personnel  requirements,  and  resource  (time/system) 
requirements  shall  not  significantly  differ  from  the 
current  available  materials  and  requirements. 

(3)  Requirement  3;  Although  this  subtask  does  not  require 

the  SLCP  criteria  and  evaluation  procedure  to  be  auto¬ 

mated,  the  possibility  of  automating  the  bookkeeping 
aspects  shall  be  considered  and  recommendations  given. 

(4)  Requirement  4;  The  SLCP  criteria  shall  encompass  three 

areas  of  activity:  procurement,  development  contractor, 
and  operation  support.  The  focus  within  these  areas 

shall  be  the  software  project  and  configuration  manage¬ 
ment  process  characteristics  which  affect  software  sup- 
portability. 

5.3  GUIDELINES. 

The  guidelines  are  organized  into  the  following  subsections: 
Background,  Evaluation  Responsibility,  Evaluation  Hierarchy,  Evalua¬ 
tion  Procedure,  Evaluation  Source  Materials,  and  Evaluation 
Automation.  A  full  set  of  materials  including  all  questions, 
definitions,  procedures  and  background  is  contained  in  appendix  A. 
Other  than  page/section  numbering  and  possible  automation  considera¬ 
tions,  appendix  A  is  structured  so  as  to  be  easily  adapted  as  an 
AFOTECP  800-2  pamphlet. 

5.3.1  Background. 

a.  The  software  life  cycle  process  (SLCP)  is  an  integral  part  of 
the  encompassing  system  life  cycle.  The  purpose  of  the  SLCP 
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evaluation  .Is  to  assess  those  high  level  aspects  of  the  SLOP  which 
are  believed  to  significantly  affect  the  resulting  supportablllty  of 
the  software  products  In  the  software  product  support  environment. 

b.  The  system  life  cycle  consists  of  four  major  phases:  Concept 
Exploration,  Demonstration  and  Validation,  Full-Scale  Development, 
and  Production  and  Deployment.  The  procurement  process  Is  typically 
emphasized  In  the  first  three  phases  of  the  system  life  cycle  and 
repeated  in  the  fourth  phase.  Production  and  Deployment,  as  required 
throughout  a  system's  operational  life.  The  first  three  phases  each 
culminate  In  a  major  decision  that  Is  marked  with  a  milestone.  These 
milestones  have  attainment  criteria  which  must  be  satisfied  before 
proceeding  to  the  next. 

c.  The  system  procurement  strategy  may  cause  system  developments 
to  skip  phases  or  to  have  various  life  cycle  activities  In  any  or  all 
phases.  If  a  phase  Is  skipped,  attainment  criteria  for  that  phase 
must  be  accomplished  during  the  previous  phase. 

d.  For  systems  containing  computer  resources,  the  system  life 
cycle  phases  entail  the  following  activities: 

(1)  Concept  Exploration  -  Explore  the  role  of  computer 
resources  within  the  system  and  plan  for  computer 
resources  within  the  system  life  cycle. 

(2)  Demonstration  and  Validation  -  Define  system  require¬ 
ments,  Including  those  allocated  to  computer  resources, 
and  determine  the  feasibility  of  alternative  computer 
resource  approaches  to  achieving  the  required  operational 
and  support  capability. 

(3)  Full-Scale  Development  -  Contract  for  and  manage  the 
development  of  computer  resources  (operational  and 
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support),  including  computer  hardware  and  software,  and 
determine  their  suitability  for  production. 

(4)  Production  and  Deployment  -  Deliver  systems  containing 
computer  resources  to  their  operational  site($}  and 
support  computer  .hardware  and  software  during  the 
system's  operational  life. 

e.  The  computer  software  development  cycle  consists  of  six  ' 

activities:  requirements  analysis,  preliminary  design,  detailed 

design,  coding  and  unit  testing.  Computer  Software  Component  (CSC) 
Integration  and  testing,  and  Computer  Software  Configuration  Item 
(CSCI)  level  testing.  These  activities  include  corresponding 
reviews,  products,  and  baselines.  Whenever  computer  software  Is 
developed,  the  corresponding  activities,  reviews,  products,  and  base¬ 
lines  are  applicable;  these  activities  may  be  repeated  If  the  soft¬ 
ware  Is  redeveloped  or  modified  during  any  phase  of  the  system  life 
cycle.  The  activities  can  occur  sequentially,  can  overlap  In  time, 
or  can  proceed  concurrently.  In  the  latter  case,  different  portions 
of  the  software  are  developed  in  parallel,  each  portion  proceeding 
sequentially  through  the  six  activities. 

f.  Although  computer  software  development  typically  occurs  In  the 
Full-Scale  Development  Phase,  It  may  also  occur  during  other  phases. 
For  example,  Concept  Exploration  may  require  the  development  of  a 
computer  software  model.  Demonstration  and  Validation  may  involve  the 
development  of  computer  software  for  a  prototype  system,  and 
Production  and  Deployment  may  necessitate  development  of  a  major  new 
computer  software  capability  in  order  to  support  an  evolving  system 
requirement.  In  fact,  it  is  common  for  the  system  life  cycle  to 
entail  computer  software  development  In  several  phases.  The  procure-  . 
ment  of  computer  software  for  a  defense  system  must  be  an  Integral 
part  of  the  system  process.  However,  procurement  events  for  computer 
software  may  not  occur  at  the  same  time  as  those  for  other  components 
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of  the  defense  system..  This  condition  may  occur,  for  example,  when 
computer  software  Is  developed  or  modified  during  more  than  one  phase 
of  the  system  life  cycle.  Typically,  computer  software  is  a  major 
computer  resource  element  and  may  be  managed  according  to  a  different 
schedule.  For  this  reason;  the  computer  software  development  cycle 
must  be  viewed  In  the  context  of  the  system  life  cycle. 

g.  The  procurement  activity  and  the  quality  of  the  resulting 
procurement  documents  and  product  reviews  will  greatly  affect  the 
supportablllty  of  the  resulting  software  products.  Within  the  pro* 
curement  activity,  the  software  development  activity  and  the  transi¬ 
tion  from  development  to  operation  support  will  greatly  affect  the 
supportablllty  of  the  resulting  software  products.  The  planning  for 
and  design  of  the  software  operation  support  life  cycle  process  will 
greatly  affect  the  eventual  software  supportablllty. 

h.  Within  the  three  activities  of  procurement,  development,  and 
operation  support  there  are  two  major  factors  which  affect  software  . 
supportablllty.  These  software  life  cycle  process  factors  are  soft¬ 
ware  project  management  and  software  configuration  management. 

1.  For  the  procurement,  development,  and  operation  support 
activities  the  underlying  characteristics  of  the  two  major  factors 
differ  somewhat.  For  example,  a  procurement  project  manager,  devel¬ 
opment  contractor  project  manager,  and  support  block  release  project 
manager  have  different  responsibilities.  However,  each  of  these 
managers  can  affect  the  software's  supportablllty. 

j.  Within  the  major  factors  of  project  management  and  configura¬ 
tion  management,  consistent  lower  level  characteristics  are  needed 
across  each  of  the  procurement,  development  contractor  and  operation 
support  activities.  Project  management  requires  extensive  planning, 
a  good  organizational  structure,  use  of  consistent  desigh/code/test 
methods,  and  well  defined  external  organizational  Interfaces. 
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Configuration  management  encompasses  the  functional  tasks  of  identi¬ 
fication  of  products,  control  of  the  process  by  which  modifications 
to  the  products  are  approved  and  implemented,  account  information  to 
help  understand  status  of  approved/unapproved  modification  requests, 
and  review  of  the  complete  configuration  management  process  and 
records  to  assure  that  configured  software  baselines  are  being 
properly  managed. 

k.  The  criteria  for  assessing  the  SLCP  impact  upon  the  support- 
ability  of  a  particular  set  of  software  products  have  been  derived 
from  the  considerations  discussed  above.  Current  OoD  initiatives 
which  are  likely  to  affect  the  form  and  content  of  these  criteria 
include  the  Ada  language  development  (MIL-STD-1815A,  refer¬ 
ence  1.4.11),  the  Defense  System  Software  Development  Standard 
(DoD-STD-2167,  reference  1.4.12),  the  Software  Technology  for 
Adaptable,  Reliable  Systems  (STARS,  reference  1.4.13),  the  updated 
DoD  Test  and  Evaluation  Policy  (DoDD  5000.3,  reference  1.4.14)  and 
the  associated  guidelines  for  software  test  and  evaluation  (refer¬ 
ence  1.4.15)  produced  by  the  Software  Test  and  Evaluation  Project 
(STEP).  These  Initiatives  along  with  current  practices  are  the 
foundation  for  defining  specific  evaluation  characteristics. 

5.3.2  Evaluation  Responsibility. 

a.  The  Software  Test  Manager  (STM)  and,  when  assigned,,  the  Deputy 
for  Software  Evaluation  (DSE)  are  the  core  of  AFOTEC  personnel 
responsible  for  assuring  that  an  adequate  software  supportability 
evaluation  is  conducted.  Their  responsibilities  vary  from  early 
Involvement  in  advance  planning  for  OTIiE  to  attending  development 
activity  key  reviews  such  as  PDR,  CDR,  FCAs,  and  various  working 
group  meetings.  These  personnel  are  responsible  for  collecting  DT&E 
software  development  information  which  can  be  used  during  OTAE,  and 
as  the  field  test  team  resident  software  experts.  It  is  their 
responsibility  to  make  sure  the  OT&E  of  the  software  is  properly 
addressed  throughout  the  software  life  cycle. 
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b.  Because  of  the  close  association  of  the  STM/DSE  with  the 
overall  software  procurement  development  and  support  processes,  it  is 
recommended  that  these  personnel  have  the  responsibility  for  assuring 
that  the  SLOP  evaluation  is  properly  conducted.  This  means  the 
STM/DSE  must  either  complete  the  SLOP  questionnaire,  or  make  sure 
other  qualified  personnel  complete  the  questionnaire.  In  most  cases 
it  is  envisioned  that  the  necessary  information  to  complete  an  SLOP 
evaluation  will  come  from  procurement  documents  (such  as  the  SON, 
PMO,  RFP,  TEMP,  CRISP,  0/S  CMP,  CRLCMP  and  working  group  documents), 
development  project  reviews  (such  as  PDR,  CDR,  FCA,  PCA),  and  soft- 
'ware  support  resources  documents  (such  as  the  CRISP,  0/S  CMP,  project 
and  c^figuratlon  management  plans).  Additional  Information  should 
be  derived  from  knowledgeable  personnel  In  the  procurement  activity 
development  contractor  and  operation  support  activity.  This  evalua¬ 
tion  information  should  be  accumulated  over  the  period  of  OT&E 
participation  In  the  system  software  life  cycle  process. 


5.3.3  Evaluation  Hierarchy 


a.  The  hierarchy  of  major  factors,  characteristics  and  lower 
level  characteristics  Is  derived  from  considerations  described  In 
section  5.3.1  and  the  requirements  of  section  5.2.  This  hierarchy  Is 
Illustrated  In  figure  5-1.  The  concept  Is  to  evaluate  at  a  high 
level  the  effect  on  software  supportablllty  of  the  software  procure¬ 
ment,  development,  and  operation/support  activities  from  a  management 
viewpoint.  These  evaluation  measures  are  partitioned  as  appropriate 
across  the  level  3  characteristics  shown  In  figure  5-1.  These 
measures  are  then  accumulated  to  the  higher  levels  to  arrive  at  a 
software  life  cycle  process  supportablllty  evaluation  measure.  This 
measure  is  then  Integrated  with  the  Software  Product  and  Software 
Support  Resources  measures  to  obtain  an  overall  software  support- 
ability  evaluation  measure. 
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b.  The  evaluation  major  factor  values  at  level  2,  along  with 
major  factor  values  from  level  2  of  the  Software  Product  and  Software 
Support  Resources  evaluations,  are  used  in  a  regression  equation  to 
determine  overall  software  supportablllty  risk.  The  major  factors 
which  are  drivers  for  this  risk  are  also  determined.  The  specific 
questions  and  guidelines  for  Interpretation  are  described  in 
appendix  A. 

5.3.4  Evaluation  Procedure.  The  SLOP  evaluation  procedure  is  an 
embedded  part  of  the  RAMSS  and  the  general  software  supportabllity 
evaluation  process  as  shown  In  figure  5*2.  The  specific  aspects  of 
the  SLCP  evaluation  are  briefly  described  In  the  following  para¬ 
graphs,  and  In  appendix  A. 

5. 3.4.1  Planning  the  Evaluation. 

a.  It  Is  necessary  for  the  STM/OSE  to  carefully  plan  *for  the 
collection  of  required  SLCP  data  In  order  to  adequately  complete  the 
SLCP  evaluation  questionnaire.  The  STM/DSE  should  review  the  SLCP 
questionnaire  during  AFOTEC  advanced  planning  along  with  the  likely 
sources  for  answers  to  the  SLCP  questions  (see  appendix  A).  A  time¬ 
table  should  be  developed  as  part  of  the  evaluation  plan  which 
specifies  when  the  Identified  source  documents  will  be  available, 
program  reviews  will  be  held,  tests  will  be  conducted,  and  key  per¬ 
sonnel  can  be  visited  to  retrieve  the  information  needed  to  answer 
the  questions  during  the  "official"  conduct  of  the  evaluation. 
Furthermore,  problems/concerns  noted  by  AFOTEC  personnel  during  this 
planning  and  data  collection  phase  can  be  presented  to  procurement 
and  development  contractor  personnel  for  possible  early  life  cycle 
resolution. 

b.  This  SLCP  focus  during  the  OTIE  planning  process  does  not 
cause  a  significant  change  in  the  usual,  process.  Most  source 
resources  for  question  resolution  will  be  similar  from  system  to 
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system.  For  the  most  part,  the  SLCP  questionnaire  can  serve  as  a 
simple  checklist  for  AFOTEC  SLCP  concerns  addressed  during  0T4E 
planning  and  software  system  data  collection  processes. 

c.  During  the  data  collection  It  should  also  be  possible  for 
the  STM  and  OSE  to  establish  a  perspective  on  the  possible  ranges  for 
the  User/Supporter  Baseline  Estimate  data.  For  example,  Information 
from  the  contractor  development  activity  should  Indicate  how  often 
changes  are  being  made  to  the  development  baseline  and  the  nature  of 
those  changes  (type,  complexity).  The  trend  over  time  for  these 
change  data  and  the  number  and  skill  level  of  the  development 
contractor  personnel  will  be  an  Indicator  of  the  associated  data 
required  for  the  User/Supporter  Baseline  Estimate  (see  section  VI). 

5. 3. 4. 2  Conducting  the  Evaluation.  Conducting  the  SLCP  evaluation 
consists  of  the  formal  completion  of  the  SLCP  questionnaire  responses 
by  the  STM,  OSE  and  other  designated  personnel.  Previously  collected 
data,  updated  so  as  to  reflect  current  software  life  cycle  process 
status,  can  be  used  as  a  basis  for  the  responses.  The  User /Supporter 
Baseline  Estimate  can  be  compared  to  associated  change  data  and 
personnel  requirements  during  development  to  help  determine  the  over¬ 
all  Impact  of  the  software  maturity  upon  the  software's  support- 
ability.  Once  the  question  responses  have  been  completed,  the 
responses  are  entered  Into  the  RAMSS  automated  support  tool  data  base 
for  analysis. 

5. 3. 4. 3  Analyzing  Evaluation  Results. 

a.  The  RAMSS  automated  support  tool  provides  SLCP  analysis 
results  In  the  form  of  evaluation  averages  at  each  level  of  the 
hierarchy,  percentile  of  the  SLCP  evaluation  scores  relative  to  all 
evaluation  data  base  SLCP  evaluation  scores,  and  the  relative  Impact 
of  the  SLCP  major  evaluation  factors  (project  management  and  configu¬ 
ration  management)  upon  the  overall  software  supportablllty  risk 
assessment. 
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b.  The  specific  interpretation  and  form  of  the  SLCP  analysis 
results  as  part  of  the  RAMSS  analysis  are  described  in  more  detail  in 
the  RAMSS  User's  Handbook  (reference  1.4.6). 

5. 3. 4. 4  Reporting  Results  of  Evaluation.  The  SLCP  evaluation 
results  are  reported  as  part  of  the  overall  RAMSS  results.  The  form 
of  these  results  is  dependent  upon  AFOTEC  reporting  requirements. 
The  output  from  the  RAMSS  automated  support  tool  forms  a  basis  for 
the  reporting  of  these  results  as  described  in  the  RAMSS  User's  Hand¬ 
book  (reference  1.4.6). 

5.3.5  Evaluation  Source  Materials.  A  summary  of  potential  source 
resources  for  use  in  the  SLCP  evaluation  Is  presented  in  this 
section.  There  is  some  overlap  In  the  use  of  resources  across 
process  activities  and  evaluation  factors.  A  list  of  source 
resources  is  shown  in  figure  5-3.  These  source  resources  are  meant 
to  be  reasonably  thorough,  but  not  necessarily  complete.  Guidelines 
for  use  of  these  materials  for  each  question  are  presented  in 
appendix  A. 

5.3.6  Evaluation  Automation. 

a.  The  evaluation  procedure  described  in  section  5.3.4  and  in 
more  detail  in  appendix  A  is  essentially  manual.  Since  there  is  only 
one  questionnaire  and  a  reasonably  finite  set  of  questions,  it  is  not 
too  difficult  to  manually  average  the  low  level  evaluation  metrics  to 
compute  the  evaluation  metrics  at  level  3  of  the  SLCP  h-ierarchy. 
However,  it  is  inconvenient  and  does  not  allow  for  various  subgroup 
computation. 

b.  It  would  be  very  simple  to  create  a  data  base  of  questions  and 
requirement  statements  which  could  be  used  as  an  automated  aid  for 
the  SLCP  evaluation  bookkeeping  functions. 
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Directives,  Regulations,  Standards 


1.  OoOO  5000.1,  Major  System  Acquisition,  19  Nov  19Sb. 

2.  OoDI  5000.2,  Major  System  Acquisition  Procedures,  19  Nov 
1985. 

3.  OoOO  5000.3,  Test  and  Evaluation,  26  Dec  1985  (DRAFT). 

4.  OoOO  5000.3  M-3,  Software  Test  and  Evaluation  Manual,  Oct 
1985. 

5.  OoOO  5000.29,  Management  of  Computer  Resources  in  Major 
Systems,  26  Apr  1976  (In  Revision). 

6.  OoOO  5000.31,  Higher  Order  Programming  Language  (HOL) 
Standardization  Policy  for  Embedded  Computers,  10  Jun 
1983. 

7.  AFR  800-14  Vol.  I,  Management  of  Computer  Resources  In 
Systems,  12  Sep  1975. 

8.  AFR  800-14  Vol.  II,  Acquisition  and  Support  Procedures 
for  Computer  Resources  In  Systems,  26  Sep  1975. 

9.  AFR  55-43,  Management  of  Operational  Test  and  Evaluation, 
28  Jun  1985. 

10.  AFR  65-3,  Configuration  Management,  1  Jul  1974. 

11. -  AFR  80-14,  Test  and  Evaluation,  12  Sep  1980. 

12.  AFR  800-4,  Transfer  of  Program  Management  Responsibility, 
15  Jun  1982. 

13.  AFSCP  800-48,  Software  Management  Indicators,  9  Dec  1985. 

14.  AFOTECR  55-1,  AFOTEC  Operations  Regulation,  1  Jun  1985. 

15.  DoO-STD-2167,  Defense  System  Software  Development,  4  Jun 
1985. 

16.  '  OoD-STO-2168,  Software  Quality  Evaluation,  24  Apr  1985 

(DRAFT). 


Figure  5-3.  SLCP  Evaluation  Source  Resources 
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17.  Do0-ST0-480A,  Configuration  Control  -  Engineering 
Changes,  Deviations  and  Waivers,  12  Apr  1978. 

18.  DoO-STD-482A,  Configuration  Status  Accounting  Data 

Elements  and  Related  Features,  1  Apr  1974. 

19.  MIL-ST0-483A,  Configuration  Management  Practices  for 
Systems,  Equipment,  Munitions,  and  Computer  Programs, 
4  Jun  1985. 

20.  MIL*STD-490A,  Specification  Practices,  4  Jun  1985. 

21.  MIL-STO*1521B,  Technical  Reviews  and  Audits  for  Systems, 
Equipments,  and  Computer  Software,  4  Jun  1985. 

22.  MIL-S-52799A,  Software  Quality  Assurance  Program 

Requirements,  1  Aug  1979. 

23.  Joint  Regulation,  Manaoement  of  Computer  Resources  In 
Defense  Systems,  30  Dec  1983  (DRAFT). 

24.  Joint  Regulation,  Data  Item  Descriptions,  4  Jun  1985. 

25.  ,  TRW  Guidebook  Series,  An  Air  Force  Guide  to  Computer 
'  Program  Configuration  Management,  Aug  1977. 

26.  POWER  System  Manager's  Manual,  1983. 

27.  ANSI/IEEE  Std  828-1983,  Software  Configuration  Management 

Plans,  23  Jun  1983. 

28.  ANSI/IEEE  Std  829-1983,  Software  Test  Documentation, 
19  Aug  1983. 


Project  Specific  Documents 


1.  Program  Management  Directive  (PMD) 

2.  Program  Management  Plan  (PMP) 

3.  Test  and  Evaluation  Master  Plan  (TEMP) 

4.  Computer  Resources  Life  Cycle  Management  Plan  (CRLCMP) 


Figure  5-3.  SLCP  Evaluation  Source  Resources  (Continued) 
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5.  Computer  Resources  Integrated  Support  Plan  (CRISP) 

6.  Operational/Support  Configuration  Management  Procedures 
(0/S  CMP) 

7.  Development  Test  and  Evaluation  Plans 

8.  Operational  Test  and  Evaluation  Plans 

9.  Contractor  Computer  Program  Development  Plan  (CPDP) 

10.  Contractor  Software  Configuration  Management  Plan  (SCMP) 

11.  Contractor  Software  Quality  Assurance  Plan 


Figure  5*3.  SLCP  Evaluation  Source  Resources  (Concluded) 


THE  BDM  CORPORATION 


BDM/ABQ-86-0090-TR 


c.  For  the  present,  it  is  convenient  to  integrate  the  SLCP 
evaluation  metric  entry  and  computation  at  the  lowest  level  of  the 
hierarchy  into  the  RAMSS  data  base  (reference  1.4.6).  Although  this 
is  not  consistent  with  the  level  3  data  entry  to  RAMSS  for  the  Soft¬ 
ware  Product  and  Software  Support  Resources  evaluations,  it  does 
automate  the  SLCP  evaluation  computations.  In  addition,  it  does  not 
preclude  entry  pf  SLCP  evaluation  results  at  the  level  3  if  another 
automated  bookkeeping  (or  alternate  set  of  low  level  characteristics) 
Is  implemented  (e.g.,  an  ASSET-SLCP  data  base). 
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SECTION  VI 

USER/SUPPORTER  BASELINE  ESTIMATE  GUIDELINES 
6.1  INTRODUCTION. 

a.  The  User/Supporter  Baseline  Estimate  Is  derived  from  a 
combination  of  historical  data  In  the  RAMSS  data  base,  and  the 
experience  of  the  system's  Using  Command/Supporting  Command  person¬ 
nel.  The  Estimate  summarizes  the  general  resources  and  level  of 
support  activity  required  to  maintain  the  subject  software  system. 
Specifically*  the  Estimate  consists  of  system  software  Identification 
Information*  block  release  cycle*  direct  personnel  full  time  equiva¬ 
lents  and  skill  level*  and  the  estimated  change  request  profile  over 
the  first  few  (up  to  three)  block  releases. 

b.  The  User/Supporter  Baseline  Estimate  derivation  Is  dependent 
upon  cooperation  from  the  system's  Using  and  Supporting  Command 
personnel.  This  cooperation  and  the  consequent  discussions  concern¬ 
ing  the  support  requirements  may  be  the  most  valuable  result  of  the 
derivation.  Typically  the  steps  to  deriving  an  Estimate  Include: 

(1)  Prepare  draft  Estimate  from  RAMSS  data  base  using  similar 
systems  and  best  guess  from  CRISP  Information 

(2)  Obtain  feedback  from  Supporting  Command  personnel  on 
draft 

(3)  Obtain  feedback  from  Using  Command  personnel  on  draft  and 
Supporting  Command  comments 

(4)  Prepare  update  to  draft 

(5)  Iterate  steps  2, '3,  4  as  often  as  necessary. 
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c.  The  derivation  of  the  User/Supporter  Baseline  Estimate  should 
be  completed  prior  to  the  software  supportablllty  evaluation.  The 
Estimate  Is  used-  as  part  of  the  evaluator  callbratlon/orlentatlon 
part  of  the  Software  Product,  Software  Support  Resources,  and  Soft¬ 
ware  Life  Cycle  Process  evaluations..  An  example  of  a  User/Supporter 
Baseline  Estimate  Is  presented  In  figure  6-1  along  with  the  estimated 
risk  computed  by  the  RAMSS  automated  support  tool. 

6.2  REQUIREMENTS. 

The  derivation  and  use  of  the  User /Supporter  Baseline  Estimate 
Is  a  unique  feature  of  the  RAMSS.  The  Impact  of  this  Estimate  upon 
the  current  Software  Supportablllty  Evaluation  Is  subject  to  several 
constraints  and/or  requirements  listed  below. 

(1)  Requirement  1.  The  RAMSS  requires  the  derivation  and  use 
of  a  User/Supporter  Baseline  Estimate.  It  Is  not  a 
requirement  as  to  how  this  Estimate  Is  derived 

(2)  Requirement  2.  It  must  be  possible  to  derive  the  data 
for  a  draft  User/Supporter  Baseline  Estimate  even  If  the 
Using  and  Supporting  Command  personnel  are  not  coopera¬ 
tive  or  cannot  agree 

(3)  Requirement  3.  The  Impact  on  0T4E  personnel  resources 
for  the  derivation  and  use  of  a  User/Supporter  Baseline 
Estimate  must  be  minimal.  The  procedure  for  deriving 
such  an  Estimate  must  Integrate  naturally  with  the 
current  AFOTEC  OT&E  process 

(4)  Requirement  4.  There  must  be  a  way  to  create,  store,  and 
retrieve  a  User /Supporter  Baseline  Estimate  as  part  of 
the  RAMSS  data  base  and  automated  tool  support.  An 
Estimate  report  for  use  during  evaluations  must  be  able 
to  be  generated  from  the  automated  tool  support  for 
RAMSS. 
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EVALUATION  REPORT  A1 :  USER/SUPPORTER  lASEUNE  CONar 
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Figure  6-1.  ExsMple  Uscr/Supporter  Biselinc  Estliate 
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6.3  GUIDELINES. 

The  guidelines  for  deriving  the  User/Supporter  Baseline 
Estimate  and  integrating  its  use  into  the  software  supportability 
evaluations  are  described  in  the  following  subsections. 

6.3.1  Derivation  of  User/Suoporter  Baseline  Estimate.  The  steps  in 
the  derivation  of  a  User/Supporter  Baseline  Estimate  are  illustrated 
in  figure  6-2  and  described  in  the  following  paragraphs. 

6. 3. 1.1  STEP  1;  Generate  Draft  Version  Using  RAMSS.  The  automated 
tool  support  for  RAMSS  (reference  1.4.6)  provides  the  capability  to 
derive  an  initial  draft  version  of  the  User/Supporter  Baseline 
Estimate  from  historical  data  and  direct  input.  The  options  include: 

(1)  Enter  data  as  determined  from  discussions  with  the  Using 
and  Supporting  Command  personnel*  CRISP  or  other 
documentation,  and/or  from  best  guess 

(2)  Select  baseline  estimate  data  from  an  entry  in  the  main¬ 
tenance  release  data  base 


(3)  Select  baseline  estimate  data  from  the  average  of  current 
maintenance  release  data  (all  or  restricted  to  the 
systems  of  the  same  software  type  as  the  subject  system) 


(4)  Select  basel  ine  estimate  data  from  among  the  current 
baseline  agreement  entries. 


The  data  to  be  derived/entered  includes: 


(1)  System  Identification  Data:  system  name,  software  system 
name,  software  system  type.  Using  Command,  and  Supporting 
Command/support  contractor  name 
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(2)  Software  Support  Concept:  block  release  duration  and  any 
overlap  in  releases,  number  of  full  time  equivalent 
(number  of  persons  times  percentage  dedicated)  support 
persons,  and  average  skill  level  of  personnel 

(3)  Baseline  Change  Profile:  total  number  of  changes,  number 
of  changes  by  type,  number  of  changes  by  complexity,  and 
number  of  changes  by  priority,  as  expected  for  up  to 
three  block  releases. 

A  "change**  is  a  formal  software  change  request  which  results  in  a 
modification  to  a  block  release  If  accepted  for  Implementation. 

6. 3. 1.2 

a.  The  current  draft  version  of  the  User/Supporter  Baseline 
Estimate  should  be  discussed  with  the  Supporting  Command  personnel. 
This  discussion  could  be  during  an  onsite  visit  by  AFOTEC  personnel, 
an  onsite  visit  by  Supporting  Command  personnel  to  AFOTEC,  a  part  of 
a  regularly  scheduled  project  working  group  meeting  (e.g.,  the 
Computer  Resources  Working  Group  (CRW6)),  or  simply  through  phone 
conversations. 


STEP  2;  Obtain  Update  to  Draft  Version  from 
Supporting  Command J 


b.  It  should  be  emphasized  that  these  baseline  estimates  are 
very  dynamic  and  represent  a  combination  of  historical  data  and  an 
educated  but  subjective  estimate  based  upon  knowledge  about  the 
actual  software  system.  Initiation  or  expanded  continuation  of  dis? 
cuss  ion  among  Using  Command,  Supporting  Command,  OT&E,  and  OT&E  per¬ 
sonnel  is  a  very  important  by-product  of  this  derivation  process. 
Much  of  the  necessary  information  should  already  exist  in  the  CRISP 
or  other  planning  documents. 
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c.  This  step  may  be  iterated  as  often  as  necessary  until  the 
Supporting  Command  personnel  are  reasonably  comfortable  with  the 
Baseline  Estimate. 


6. 3. 1.3  STEP  3;  Obtain  Update  to  Draft  Version  from  Using  Command. 


a.  The  current  draft  version  and  any  Supporting  Command  updates 
should  be  discussed  with  the  Using  Command  personnel.  This  process 
Is  very  similar  to  STEP  2  with  the  exception  that  the  Using  Command 
personnel  will  tend  to  have  a  more  general  view  of  the  support 
resource  requirements.  The  block  release  duration,  total  number  and 
skill  level  of  personnel,  and  total  number  of  change  requests  per 
block  release  should  be  discussed.  More  detailed  Information  such  as 
type,  complexity,  and  priority  of  changes  are  likely  to  be  only 
lightly  discussed.  Occasionally  the  priority  of  changes  will  be 
discussed. 


b.  The  same  emphasis  and  Iteration  as  in  STEP  2  Is  appropriate 
for  this  STEP  3. 

6. 3. 1.4  STEP  4;  Update  Draft  Version  Using  RAMSS.  Using  the  same 
automated  support  tools  as  In  STEP  1,  the  draft  version  of  the  User/ 
Supporter  Baseline  Estimate  should  be  updated.  The  updated  version 
can  be  printed  as  a  report,  and  If  necessary  redistributed  to  the 
Supporting  and  Using  Command  personnel  for  further  comments  and 
Iteration.  After  an  appropriate  series  of  updates  and  Iterations, 
there  should  be  some  consensus  of  the  Estimate.  This  Estimate  can 
then  be  used  to  obtain  the  initial  estimate  of  the  software  support- 
ability  risk  and  as  part  of  the  software  supportablllty  evaluation. 


6.3.2  Use  of  User/Supporter  Baseline  Estimate. 


a.  The  User/Supporter  Baseline  Estimate  is  used  for  several 
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(1)  Catalyst  for  indepth  discussions  of  software  support 
resources  and  life  cycle  process  management  requirements 

(2)  Information  for  use  during  the  software  supportability 
evaluations,  primarily  for  evaluator  orientation. 


b.  The  first  use  of  the  Estimate  as  described  above  is  the  most 
important.  Without  the  discussion,  the  important  support  resource 
issues  would  not  come  to  be  known,  and  a  valid  Estimate  would  be 
unlikely.  This  knowledge  allows  for  more  precise  personnel  require¬ 
ments  to  be  specified,  for  a  better  understanding  of  the  block 
release  cycle,  and  a  more  realistic  set  of  change  profile  counts. 
The  discussion  also  stimulates  a  better  understanding  of  the  defi¬ 
ciencies,  benefits  and  major  Issues  In  the  project  and  configuration 
management  life  cycle  processes,  all  of  which  should  be  reflected  In 
the  CRISP/CRLCMP  and/or  0/S  CMP. 

c.  The  Estimate  is  Input  to  the  Software  Product,  Software 
Support  Resources,  and  Software  Life  Cycle  Process  evaluations  so  as 
to  provide  additional  system  knowledge  to  aid  an  evaluator's 
responses  based  upon  expected  software  maintenance  activity  and 
resource  requirements. 

d.  The  Estimate  can  be  derived  without  Using/Supporting  command 
participation,  especially  when  early  system  software  supportability 
assessments  are  derived.  However,  due  to  the  benefits  derived  from 
command  participation,  'it  is  highly  recommended. 
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APPENDIX  A 

SOFTWARE  LIFE  CYCLE  PROCESS  EVALUATOR'S  GUIDE 

The  purpose  of  this  eppendix  Is  to  provide  the  Software  Test 
Manager  (STM)  and  Deputy  for  Software  Evaluation  (DSE)  with  the 
Information  needed  to  accomplish  the  Air  Force  Operational  Test  and 
Evaluation  Center's  (AFOTEC's)  software  life  cycle  process  evalua¬ 
tion.  In  this  appendix,  "software  life  cycle  process"  Is  limited  In 
scope  to  software  project  management  and  software  configuration  man¬ 
agement  assessments. 

This  appendix  Is  an  evolutionary  document  that  should  be 
updated  periodically.  Questions  contained  are  Intended  to  be  only  a 
representative  sample  of  software  life  cycle  process  questions.  This 
version  represents  tn  Initial  version  and  your  written  Inputs  are 
solicited  for  the  n^^t  update. 

This  appendix  Is  Intended  to  be  a  volume  In  a  series  of 
Software  Operational  Test  and  Evaluation  Guidelines  prepared  by  the 
Software  Evaluation  Division  of  the  Logistics  Directorate.  It  Is 
Intended  for  use  In  the  operational  test  and  evaluation  of  software. 
Comments  should  be  directed  to  the  Office  of  Primary  Responsibility 
(OPR).  The  series  of  guidelines  are: 

AFOTEC  Pamphlet  800-2,  Volume  I  -  Management  of  Software  Opera¬ 
tional  Test  and  Evaluation 

AFOTEC  Pamphlet  800-2,  Volume  2  -  Reserved 

AFOTEC  Pamphlet  800-2,  Volume  3  -  Software  Maintainability  - 

Evaluator's  Guide 

AFOTEC  Pamphlet  800-2,  Volume  4  -  Software  Operator-Machine 

Interface  -  Evaluator's  Guide 
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AFOTEC  Pamphlet  800-2,  Volume  5  •  Software  Support  Facility 

Evaluation  -  User's  Guide 
(currently  not  being 
published) 

AFOTEC  Pamphlet  800-2,  Volume  6  -  Reserved 
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A.l  GENERAL. 

a.  Software  supportability  Is  a  measure  of  the  adequacy  of 
personnel,  resources,  and  procedures  to  facilitate  the  support 
activities  of  modifying  and  Installing  software,  establishing  an 
operational  software  baseline,  and  meeting  user  requirements.  Soft¬ 
ware  supportability  Is  i  function  of  the  luallty  of  the  software 
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products,  the  capabilities  of  the  software  support  resources,  and  the 
life  cycle  management  processes  which  control  the  procurement,  devel* 
opment,  operation  and  support  of  the  software. 

b.  Tlie  focus  of  this  guide  Is  upon  the  life  cycle  management 
processes  of  software  project  management  and  software  configuration 
management. 

A.2  OVERVIEW. 

a.  The  STM/DSE  should  read  paragraphs  A.l  through  A.9  In  their 
entirety  and  understand  the  evaluation  concept  and  procedures  before 
beginning  any  evaluation.  These  pages  provide  the  evaluator  with: 

(1)  A  background  of  the  AfOTEC  software  maintainability 
evaluation  concept 

(2)  A  basic  understanding  of  the  evaluation  procedures 

(3)  Detailed  Instruction  for  using  the  software  life  cycle 
process  quest lonnair els. 

b.  Attachment  A1  contains  the  questionnaires  and  explanatory 
Information  on  each  question.  This  Information  Is  provided  In  an 
attempt  to  ensure  that  the  STM/DSE  fully  understands  the  Intent  of 
each  question.  Included  are  definitions  of  terms,  examples,  explana¬ 
tions,  and  special  case  response  Instructions,  as  necessary.  Attach¬ 
ment  A1  Is  designed  to  be  used  as  the  source  of  questions  for  the 
evaluation. 

c.  Attachment  A2  contains  a  summary  list  of  all  the  questions  for 
quick  reference.  Attachment  A3  Is  a  glossary  of  terms. 

d.  Questions  are  not  listed  In  the  order  of  Imoortance. 
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A.3  SOFTWARE  LIFE  CYCLE  PROCESS  EVALUATION  METHOD. 

•.  Tht  method  for  evaluating  the  software  life  cycle  process  Is 
based  on  the  use  of  closed-form  questionnaires  with  optional  written 
rationales  justifying  the  evaluation  score  assigned  to  a  question. 
These  questionnaires  are  designed  to  determine  the  degree  to  which 
certain  desirable  attributes  affecting  software  supportablllty  are  or 
will  be  part  of  the  software  life  cycle  process.  The  elements  of  the 
software  life  cycle  process  and  their  relationships  are  shown  In 
figure  A-1  and  are  described  In  the  following  paragraphs.  The  hier¬ 
archical  evaluation  structure  shown  In  the  figure  enables  the  STM/DSE 
to  Identify  potential  software  supportablllty  problems  at  various 
levels:  category/major  factor  (project  management,  configuration 
management),  characteristics  (planning,  organizational  structure, 
design  methods.  Implementation  methods,  test  strategies,  and  project 
Interfaces),  low  level  characteristics  (Individual  questions),  or 
some  combination.  Each  question  should  be  evaluated  on  the. basis  of 
Its  characteristic. 

b.  Software  life  cycle  process  management  Is  a  combination  of  the 
policy,  methodology,  procedures,  and  guidelines  applied  In  a  software 
environment  to  the  software  development  and  support  life  cycle 
activities.  The  major  management  aspects  for  purposes  of  software 
supportablllty  can  be  grouped  Into  two  categories  or  major  factors: 
software  project  management  and  software  configuration  management. 
The  major  factor  characteristics  are  evaluated  with  respect  to  their 
Impact  upon  software  supportablllty  considerations.  '  All  life  cycle 
phases  and  the  activities  of  procurement,  development,  operation  and 
support  are  applicable. 

c.  Software  project  management  Is  concerned  with  producing,  a 
software  product:  either  the  initial  production  baseline  or  a  version 
of  the  production  baseline.  During  development  there  are  many  man¬ 
agement  characteristics  which  will  Influence  the  supportablllty  of 
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Figure  A-1.  Software  Life  Cycle  Process  Evaluation  Hierarchy 
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the  software.  Procurement  activity  Is  responsible  for  overall 
project  management  Including  planning  for  supportablllty.  Develop¬ 
ment  contractor  activity  Is  responsible  for  managing  the  delivery  of 
the  production  baseline  within  the  procurement  activity  requirements. 
During  post  deployment  the  software  support  activity  directly 
controls  the  baseline  update  process.  Lack  of  planning,  poor 
organizational  structure,  Inadequate  des1gn/1^>1ementat1on/test 
methods  and  strategies,  and  poor  Interfaces  among  responsible 
activities  all  affect  the  supportablllty  of  the  resulting  software 
product. 

d.  Software  configuration  management  Is  concerned  with  providing 
a  means  through  technical  and  administrative  direction  and  surveil¬ 
lance  to: 

.  (1)  Identify  and  document  the  functional  and  physical  char¬ 
acteristics  of  a  configuration  Item, 

(2)  Control  changes  to  those  characteristics,  and 

(3)  Record  and  report  change  processing  and  Implementation 
status. 

The  three  areas  that  produce  these  results  are  configuration  Identi¬ 
fication,  configuration  control,  and  configuration  status  accounting. 
A  fourth  area,  configuration  audits,  verifies  that  a  completed 
product  and  Its  documents  meet  contractual  requirements.  The 
procurement,  development,  operation  and  support  activities  all  have 
configuration  management  responsibilities  to  assure  that  the  baseline 
production  products  and  subsequent  revisions  are  properly  controlled. 
These  configuration -magagement  responsibilities  have  an  Impact  upon 
the  supportablllty  of  the  software  products. 
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e.  Responsibility  for  the  procurement  activity  rests  with  those 
government/mllltary  organizations  which  assure  delivery  of  a  produc¬ 
tion  system.  Primary  organizations  Include  the  cognizant  program 
office  and  the  Implementing  command.  The  supporting  command  and 
using  command  have  Important  Interfaces  with  the  primary  organiza¬ 
tions  during  the  procurement  process.  Organizations  responsible  for 
DTtE,  OT&E,  and  perhaps  IVtV  have  specific  roles  In  the  procurement 
process. 

f.  Responsibility  for  the  development  contractor  activity  rests 
with  those  government/private  organizations  which  accomplish  the  full 
scale  development  of  the  production  system.  Primary  organizations 
Include  the  contractor. 

g.  Responsibility  for  the  operation  support  activity  rests  with 
those  government/mllltary/private  organizations  which  operate/use  the 
production  system  and/or  provide  support  for  the  production  system. 
Primary  organizations  Include  the  using  command  and  supporting 
command.  Contractor  organizations  may  provide  support  for  the 
supporting  command. 


A. 3.1  Software  Project  Management  Test  Factors.  The  software 
project  management  evaluation  is  based  on  six  characteristics  or  test 
factors:  planning,  organization  structure,  design  methods,  imple¬ 
mentation  methods,  test  strategies,  and  project  interfaces. 
Definitions  of  these  test  factors  and  discussions;  of  their  applica¬ 
tion  In  the  evaluation' process  are  given  In  the  following  paragraphs. 


A. 3.1. 1  Plannint 


a.  The  software  project  management  process  utilizes  planning 
which  enhances  software  supportablllty  to  the  extent  that  plans  for 
the  development,  test,  product  transfer,  operation  and  support  exist, 
have  been  Implemented,  have  been  appropriately  coordinated  across 

ictivities',  ind  iatisfy  rantractuai  ano/or  '“guidtion  "aqui raments. 
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b.  Major  planning  documents  for  the  procurement  activity 
Include  the  Program  Management  Directive  (PM0)»  Program  Management 
Plan  (PMP),  Test  and  Evaluation  Master  Plan  (TEMP),  Computer 
Resources  Integrated  Support  Plan  (CRISP),  Operation/Support  Con¬ 
figuration  Management  Procedures  (0/S  CMP),  the  Development  Test  and 
Evaluation  (DTtE)  plans,  and  the  Operational  Test  and  Evaluation 
(OTtE)  plans.  In  the  joint  Logistics  Coemanders  Software  Standardi¬ 
zation  program,  the  CRISP  and  0/S  CMP  are  combined  Into  a  Computer 
Resources  Life  Cycle  Management  Plan  (CRLCMP). 

c.  Major  planning  documents  for  the  development  contractor 
activity  Include  the  System/Segment  Specification,  the  Software 
Development  Plan  (SOP),  Software  Configuration  Management  Plan 
(SCMP),  Software  Quality  Assurance  Plan  (SQAP),  Software  Standards 
and  Procedure  Manual  (SSPM)  and  the  Software  Test  Plan  (STP). 

d.  Major  planning  documents  for  the  operation  support  activity 
include  the  TEMP.  OTSiE  plans,  OTIE  plans,  CRISP,  0/S  CMP,  and  CRLCMP. 

e.  One  of  the  most  Important  results  of  good  planning  Is  the 
coordination  of  Information  across  the  various  planning  documents  to 
minimize  redundancy  and  satisfy  the  necessary  content  requirements  of 
the  plans.  The  conciseness  and  level  of  detail  of  planning  Informa¬ 
tion  Is  very  Important.  Frequently  plans  act  as  a  place  holder  for 
"real"  Information,  serving  a  role  little  better  than  a  "TBO".  To 
say  that  structured  prograimilng  standards  will  be  followed  Is  not 
precise  enough  In  the  Software  Standards  and  Procedure  Manual. 
Precise  programming  requirements  which  represent  the  contractor's 
definition  of  "structured  programming  standards"  must  be  specified  In 
a  manner  suitable  for  quality  assurance  testing  for  conformance.  As 
another  exeunple.  It  Is  not  satisfactory  to  Indicate  In  the  CRISP  that 
support  resources  space  requirements  are  4,800  square  feet.  It  Is 
necessary  to  Indicate  how  that  space  Is  allocated  anrang  support 
personnel  office  space,  system  support  space,  storage/library  space. 


II 
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and  iny  other  space  allocations  which  night  be  peculiar  to  the 
particular  application  system.  Furthermore,  a  top  level  facility 
layout  showing  the  physical  relationship  among  the  space  allocations 
Is  appropriate. 

f.  When  systems  have  Interservice  operability  requirements, 
then  plans  for  the  appropriate  Interservice  Interfaces  and  Joint 
activities  should  be  clearly  specified,  particularly  the  plans  for 
supporting  the  software. 

g.  When  development  contractor  activity  Involves  subcontrac¬ 
tors,  the  plans  for  managing  the  subcontractor  effort  and  the  sub¬ 
contractor  Internal  plans  for  managing  their  efforts  should  be 
clearly  specified,  particularly  any  plans  for  supporting  deliverable 
software. 


h.  A  good  plan  will  possess  a  concise  statement  of  the  objec¬ 
tives  of  the  plan,  the  techniques  and  methods  by  which  the  plan  will 
be  Implemented,  the  responsible  organizational  elements  for  making 
sure  the  plan  Is  Implemented  and  evolved  as  necessary,  the  schedule 
by  which  the  objectives  of  the  plan  are  to  be  accomplished,  and  the 
relationships  of  the  plan  to  any  other  system  elements. 

1.  Planning  Is  evaluated  for  how  well  the  life  cycle  plans 
address  software  supportabillty, 

A. 3. 1.2  Organization  Structure. 

a.  The  software  project  management  process  organization  struc¬ 
ture  enhances  software  supportabillty  to  the  extent  that  the  physical 
structure,  functional  responsibilities,  external  Interfaces  and 
assigned  personnel  provide  for  continuity  over  the  software  life 
cycle  phases,  and  have  proper  Interfaces  with  organizations  respon¬ 
sible  for  software  support. 
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b.  The  procurement  activity  must  have  an  organization  structure 
which  provides  continuity  across  all  life  cycle  phases  and  through 
each  milestone.  The  organization  structure  must  provide  for  adequate 
dissemination  and  coordination  of  information  among  all  activities. 
Organization  elements  must  provide  functions  for  project  oversight, 
configuration  management,  quality  evaluation,  project  reviews  and 
audits,  testing  and  evaluating  transfer  of  responsibility,  and  plans 
and  policies. 

c.  The  development  contractor  activity  must  have  an  organiza¬ 
tion  structure  which  matches  the  work  breakdown  structure  and  pro¬ 
vides  continuity  throughout  all  full  scale  development  activities  and 
the  transition  Into  post  deployment  support.  Appropriate  organi¬ 
zational  elements  should  exist  for  Internal  configuration  management, 
quality  assurance,  test  and  evaluation,  product  development,  and 
procurement/support  contractual  Interface  activity. 

d.  The  operation  support  activity  must  have  an  organization 
structure  which  satisfies  mission  requirements  within  the  require¬ 
ments  Imposed  by  the  procurement  and  development  activity  organiza¬ 
tion  and  applicable  regulations  and  directives.  Organization  ele¬ 
ments  should  be  established  early  In  the  development  phase  to  assure 
proper  transition  to  post  deployment  support  through  understanding  of 
the  software  support  requirements. 

« 

e.  Organization  structure  Is  evaluated  for  how  well  software 
supportablllty  Issues  are  able  to  be  addressed  within  the  physical 
form  of  the  structure  and  the  functional  responsibilities  of  the 
organizational  elements  using  the  assigned  personnel. 

A. 3. 1.3  Design  Methods. 

a.  The  software  project  management  process  utilizes  design 
methods  which  enhance  software  supportablllty  to  the  extent  that 
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design  methodology  standards  and  conventions  are:  1)  documented, 
followed,  and  validated  through  quality  assurance,  2)  can  be  transi* 
tioned  to  support  activity,  and  3)  produce  adequate  design  specifics^ 
tions  which  reflect  supportablllty  characteristics. 

b.  The  procurement  activity  design  methods  are  reflected  In  the 
requirements  Imposed  upon  the  development  contractor  activity  through 
the  system/segroent  specification  and  the  request  for  proposal. 
Procurement  monitoring  of  development  contractor  design  activities 
and  acceptance  of  those  activities  Is  also  a  reflection  of  the  pro¬ 
curement  activity  design  methods. 

c.  The  development  contractor  activity  design  methods  should  be 
defined  In  an  Internal  standards  and  convention  manual,  and  validated 
by  a  quality  assurance  function.  The  methods  should  reflect  use  of  a 
consistent  methodology,  traceability  between  requirements  and  produc¬ 
tion  products,  traceability  of  design  decisions.  Information  hiding, 
and  use  of  techniques  to  enhance  the  software  product  characteristics 
of  modularity,  descriptiveness,  consistency,  simplicity,  expandabil¬ 
ity,  and  Instrumentation.  Automated  tool  support  as  an  aid  to  devel¬ 
opment  design  and  support  design  evolution  Is  an  Important  part  of 
the  development  contractor  design  methods. 

d.  The  operation  support  activity  design  methods  should  be 
defined  at  a  high  level  In  a  procurement  activity  requirements  speci¬ 
fication,  and  at  a  lower  level  by  an  Internal  support  standards  and 
conventions  manual.  The  methods  should  have  a  close  similarity  to 
the  methods  used  by  the  development  contractor  activity  In  order  to 
facilitate  transition  of  the  software  design  evolution  to  the  support 
activity. 

e.  Design  methods  are  evaluated  for  characteristics  which 
Indicate  that  software  supportablllty  has  been  designed  Into  the 
software  products. 
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A.3.1  Implenientatlon  Methods. 

a.  The  software  project  management  process  utilizes  Implementa¬ 
tion  methods  which  enhance  software  supportablllty  to  the  extent  that 
Implementatlon/coding/testing  methodology,  standards,  and  conventions 
are:  1)  documented,  followed,  and  validated  through  quality 
assurance,  2)  can  be  transitioned  to  the  support  activity,  and 
3)  produce  supportable  production  products. 

b.  The  procurement  activity  Implementation  methods  are  reflected 
In  the  requirements  Imposed  upon  the  development  contractor  activity 
for  Implementatlon/coding  standards  and  the  process  through  which 
such  standards  and  the  form  of  the  production  products  are  reviewed 
and  accepted  for  operation  and  support. 

c.  The  development  contractor  activity  Implementation  methods 

should  be  defined  In  an  Internal  standards  and  conventions  manual  and 
validated  by  a  quality  assurance  function.  The  methods  should  . 

reflect  use  of  acceptable  Implementation  team  organizational  strate¬ 
gics  such  as  the  chief  programmer  team  methods  which  enhance  trace¬ 
ability  among  requirements/design/product,  and  techniques  to  enhance 
the  software  product  characteristics  of  modularity,  descriptiveness, 
consistency,  simplicity,  expandability,  and  Instrumentation.  Auto¬ 

mated  tool  support  as  an  aid  to  development  Implementation  and  change 
processing  Is  an  Important  part  of  the  development  contractor  Imple¬ 
mentation  methods. 

d.  The  operation  support  activity  Implementation  methods  should 
be  defined  at  a  high  level  In  a  procurement  activity  requirements 
specification  and  other  support  documents  such  as  the  CRLCMP  (or 
CRISP  and  0/S  CMP).  Specific  methods  should  be  defined  at  a  low 
level  by  an  Internal  standards  and  conventions  manual.  The  methods 
should  have  a  close  similarity  to  the  methods  used  by  the  development 
contractor  activity  In  order  to  facilitate  transition  of  the.  software 
implementation  evolution  :o  the  support  activity. 
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c.  Implementation  methods  are  evaluated  for  consistency  with 
standards,  availability  of  automated  tool  support  capabilities  1n  the 
form  of  software  benches  and  Integrated  laboratory  test  beds, 
potential  for  effective  use  during  software  support,  and  the  trace* 
ability  of  Implemented  product  status  and  top  level  requirements. 

A. 3. 1.5  Test  Strategies. 

a.  The  software  project  management  process  utilizes  test 
strategies  which  enhance  software  supportablllty  to  the  extent  that 
the  test  plans,  descriptions,  procedures,  and  results  have  been: 
1)  documented,  2)  can  be  transitioned  to  the  support  activity,  and 
3)  provide  for  a  consistent  and  systematic  process  for  verifying  and 
validating  that  software  requirements  have  been  satisfied. 

b.  The  procurement  activity  test  strategies  are  documented  In 
the  TEMP,  OTAE  plans  and  reports,  OTtE  plans  and  reports,  optionally 
In  IVtV  plans  and  reports,  the  preliminary  and  formal  qualification 
tests  (PQTs  and  FQTs),  and  the  acceptance  strategies  which  revolve 
around  formal  reviews  (e.g.,  SSR,  PDR,  CDR,  TRR)  and  audits  (e.g., 
FCA,  PCA).  The  test  strategies  should  clearly  Indicate  software  test 
objectives,  relationships  to  system  test  objectives,  relationships 
among  the  various  test  organizations  and  resultf  (e.g.  OTIE,  OT&E, 
IVliV),  contractually  binding  aspects  of  tests  such  as  the  schedule 
and  deliverables,  and  precisely  what  tests  will  constitute  acceptance 
of  the  production  product.  A  test  strategy  for  the  transition  period 
after  the  production  decision  and  for  a  defined  period  of  time  after 
PMRT  should  be  specifically  addressed. 

c.  The  development  contractor  activity  test  strategies  are 
documented  In  test  plans,  procedures  and  reports.  Automated  support 
In  the  form  of  software  benches,  laboratory  Integrated  test  beds,  and 
operational  Integrated  systems  has  a  major  Impact  upon  the  effective¬ 
ness  of  the  tests,  and  a  clear  strategy  for  use  of  such  tools  should 
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bt  documented,  used,  end  early  transitioned  to  the  support  activity. 
These  test  strategies  should  address:  1)  features  to  be  tested, 
2)  traceability  to  the  requirements  specifications,  and  3)  among  the 
various  test  documents,  a  consistent  approach  to  testing  various 
levels  (e.g.,  unit.  Integrated,  system),  environmental  requirements, 
organizational  responsibilities  and  Interfaces,  schedule,  deliver¬ 
ables,  risk  and  contingencies,  and  acceptance/approvals. 

d.  The  operation  support  activity  test  strategies  documentation 
Is  similar  to  the  procurement  activity  (e.g.,  for  the  TEMP  and  FOT&E 
plans  as  dynamic  documents)  and  the  development  contractor  activity 
(e.g.,  via  transition  of  the  test  plans/procedures/results  and  auto¬ 
mated  tools  to  the  support  activity).  Coordination  between  the 
operational  activity  and  support  activity  test  strategies  Is  Impor¬ 
tant  during  post  deployment  due  to  the  requirement  to  use  operational 
test  beds.  This  coordination  should  be  reflected  via  resource 
requirements  In  the  top  level  planning  documents  such  as  the  TEMP, 
CRLCMP  (or  CRISP  and  0/S  CMP)  and  specific  software  support  manage¬ 
ment  project  (I.e.,  block  release)  Internal  documents.  Similar  test 
strategy  characteristics  as  In  subparagraphs  c  and  d  above  should  be 
present  In  the  operation  support  activity  test  strategies. 


e.  Test  strategies  are  evaluated  for  how  well  the  strategies 
provide  for  a  delivery  of  mature  software  products  and  retest  of 
those  products  during  software  support. 


A. 3. 1.6  Project  Interfaces. 


a.  The  software  project  management  possesses  organization 
Interfaces  which  enhance  software  supportabllity  to  the  extent  that 
external  project  organization  relationships  and  responsibilities  are: 
1)  defined,  2}  provide  a  valuable  functional  role,  and  3)  contribute 
to  systematic  cost  effective  procurement,  development,  operation  and 
support  processes. 
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b.  The  procurement  activity  organization  interfaces  are 
primarily  with  the  development  conti  :«ctor  activity,  the  operation 
support  activity,  interface  working  groups  required  by  regulations 
and-  other  higher  level  groups  (e.g.  military,  DoO,  federal  government 
agencies.  Congress,  public).  An  IViV  Interface  may  also  be  required. 
It  is  necessary  that  each  of  these  Interfaces  is  defined  to  a  level 
of  detail  consistent  with  the  particular  application  system.  Func¬ 
tional  purpose  and  responsible  persons  should  be  identified. 

c.  The  development  contractor  activity  organization  Interfaces 
include  the  procurement  activity,  interface  working  groups,  and 
higher  level  internal  organization  elements  (e.g.,  corporate  manage¬ 
ment).  An  IVtV  interface  may  also  be  required.  In  addition,  if  sub¬ 
contractors  are  involved,  this  interface  must  be  clearly  established. 
Functional  purpose  and  schedule  of  contact  should  be  defined  and 
responsible  persons  identified. 

• 

d.  The  operation  support  activity  organization  Interfaces  are 
very  similar  to  those  of  the  procurement  activity. 

e.  Organization  interfaces  are  evaluated  for  their  effective¬ 
ness  in  resolving  interorganization  Issues  concerning  software 
support. 

A. 3. 2  Software  Configuration  Management  Test  Factors.  The  software 
configuration  management  evaluation  is  based  on  four  characteristics 
or  test  factors:  software  configuration  identification,  software 
configuration  control,  software  status  accounting,  and  software 
audits.  Definitions  of  these  test  factors  and  discussion  of  their 
application  in  the  evaluation  process  are  given  in  the  following 
paragraphs. 
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A.3.2.1  Software  Configuration  Identification. 

■a.  The  software  configuration  management  process  utilizes  con¬ 
figuration  Identification  which  enhances  software  supportablllty  to 
the  extent  that  the  software  documentation  properly  Identifies  the 
configuration  Items,  their  characteristics,  and  their  relationships 
according  to  required  standards  and  regulations. 

b.  The  procurement  activity  Is  responsible  for  following  existing 
guidelines  and  regulations  for  Identification  of  software  configura¬ 
tion  Items,  and  assuring  that  these  guidelines  and  regulations  are 
contractually  required  by  the  development  contracto*.  The  procure¬ 
ment  activity  Is  also  responsible  for  monitoring  contractor  use  of 
the  guidelines  and  regulations  to  assure  that  the  functional,  allo¬ 
cated,  developmental,  and  production  software  baselines  d«*e  properly 
Identified. 

c.  The  development  contractor  activity  Is  responsible  for  follow¬ 
ing  contractual  requirements  relative  to  configuration  management. 
This  should  Include  development  of  a  software  configuration  manage¬ 
ment  plan  In  which  configuration  Identification  standards  and  proce¬ 
dures  for  the  controlled  software  baselines  are  specified.  Indepen¬ 
dent  of  contractual  requirements.  Internal  configuration  Iden¬ 
tification  standards  and  procedures  should  exist. 

d.  The  operation  support  activity  Is  responsible  for  continuation 
of  the  same  configuration  Identification  requirements  as  required  for 
the  development  contractor  activity.  In  addition,  certain  monitoring 
responsibilities  of  the  procurement  activity  are  assured  by  the 
operation  support  activity.  The  CRLCMP,  CRISP,  and  0/S  CMP  are  the 
primary  operation  support  activity  software  configuration  management 
planning  documents. 
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t.  Configuration  Identification  Is  evaluated  for  how  well  the 
controlled  baselines  are  Identified,  unique  Identification  problems 
such  as  multiple  locations/version  variations  are  solved,  compliance 
with  regulations  and  standards,  and  the  use  of  automated  tools  to 
support  generation  and  update  of  configuration  Indexes  for  the  base¬ 
lines. 


A. 3. 2. 2  Software  Configuration  Control. 

a.  The  software  configuration  management  process  utilizes  con¬ 
figuration  control  which  enhances  software  supportablllty  to  the 
extent  change  decisions  to  software  baselines  are  made,  administered, 
and  Implemented. 

b.  The  procurement  activity  makes  change  decisions  to  software 
baselines  through  a  system  configuration  control  board.  Any  change 
request  to  a  functional,  allocated,  or  production  software  baseline 
must  be  approved  by  the  procurement  configuration  control  board.  The 
changes  are  administered  by  the  program  office’s  configuration 
management  organization.  Implementation  Is  generally  accomplished  by 
the  development  contractor  activity. 

c.  The  development  contractor  activity  makes  change  decisions  to 
developmental  software  baselines.  Administration  of  such  changes 
should  be  through  an  Internal  configuration  management  organization. 
Implementation  is  by  project  software  personnel.  In  addition, 
changes  to  the  functional,  allocated,  or  production  baseline  (prior 
to  PMRT)  are  implemented  by  the  development  contractor  activity. 
Interfaces  among  participating  contractors  must  be  established  to 
maintain  proper  configuration  control  of  the  developmental  products. 

d.  The  operation  support  activity  assumes  the  responsibility  to 
implement  change  requests  to  the  software  baselines  at  PMRT. 
Frequently,  some  level  of  configuration  control  Is  accomplished  by 
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th8  support  activity  prior  to  PMRT  for  «ase  In  transition.  Decisions 
for  making  changes  after  PMRT  are  shared  among  the  using  command, 
supporting  command,  any  Interservice  commands  as  appropriate,  and 
support  subcontractors  as  appropriate.  The  CRLCMP,  CRISP,  and  0/S 
CMP  are  the  primary  planning  documents  for  the  operation  support 
activity  software  configuration  management. 

e.  Configuration  control  Is  evaluated  for  how  well  chenges  to  the 
functional,  allocated,  developmental,  and  production  baselines  are 
controlled.  This  evaluation  Includes  the  adequacy  of  control  proce¬ 
dures  and  forms,  the  capability  to  transition  such  procedures  to  the 
support  activity  procedures,  the  adequacy  of  the  Interface  control 
among  the  organizations  responsible  for  some  aspect  of  configuration 
control,  and  the  used  of  automated  tools  to  protect  Inadvertent 
change  and  assist  In  administering  approved  changes. 

A.3.2.3  Software  Status  Accounting. 

a.  The  software  configuration  management  process  utilizes  status 
accounting  which  enhances  software  supportablllty  to  the  extent  that 
configuration  Identification  and  changes  to  the  configured  Items  are 
tracked  and  reported  through  a  configuration  Index  and  change  status 
reports. 

b.  The  procurement  activity  Is  responsible  for  monitoring  the 
status  of  the  baseline  development.  Status  accounting  provides  the 
procurement  activity  with  visibility  and  traceability  of  baseline 
configurations  and  their  changes.  The  program  office  configuration 
management  organization  should  maintain  official  baseline  status 
accounting  reports  for  use  by  the  system  configuration  control  board. 

c.  The  development  contractor  activity  uses  status  accounting 
Information  (configuration  Index  and  change  reports)  for  Internal 
management  visibility  and  traceability,  and  for  external  government 
.‘•eporcing  requirements. 
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d.  The  operation  support  activity  uses  status  accounting  Informa* 
tion  for  coordination  of  software  maintenance  tasks  that  may  Involve 
many  organizations  In  widely  scattered  locations^  as  well  as  for 
usual  Internal  management  visibility  and  Implementation  change 
status.  The  CRLCMP,  CRISP,  and  0/S  CMP  are  the  primary  operation 
support  software  configuration  management  planning  documents. 

e.  Status  accounting  Is  evaluated  for  how  well  the  changes  to 
software  baselines  are  tracked  and  reported,  the  capability  of  auto¬ 
mated  tools  to  support  the  tracking,  and  the  effectiveness  of  Inter¬ 
faces  for  communicating  status  accounting  Information  among  organiza¬ 
tional  elements  (e.g.,  program  office,  contractor,  supporting 
command). 


A. 3. 2. 4  Software  Configuration  Audit/Review. 

a.  The  software  configuration  management  process  utilizes  con¬ 
figuration  audits/reviews  which  enhance  software  supportablllty  to 
the  extent  that  the  functional  and  physical  configuration  of  the 
software  baselines  has  been  audited/reviewed  for  compliance  with 
contract  requirements. 

b.  The  procurement  activity  Is  responsible  for  preparation  and 
approval  of  the  formal  audits  and  reviews:  functional  configuration 
audit,  physical  configuration  audit,  and  formal  qualification  review. 

c.  The  development  contractor  activity  Is  responsible  for  prepa¬ 
ration  and  conduct/assist  of  the  formal  configuration  audits  and 
reviews.  In  addition.  Internal  configuration  audits  should  be 
periodically  done  on  developmental  baselines  to  provide  assurance 
that  the  configuration  Identification,  control,  and  status  accounting 
functions  are  being  properly  administered  and  the  resulting  configu¬ 
ration  Information  Is  consistent. 
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d.  The  operation  support  activity  Is  responsible  for  inonltorlng 
the  formal  audits  and  reviews  prior  to  PMRT  and  for  preparation  and 
conduct  of  updated  baseline  configuration  audits  and  reviews  after 
PMRT.  The  CRLCMP,  CRISP,  and  0/S  CMP  are  the  primary  operation  sup¬ 
port  activity  software  configuration  management  planning  documents. 

e.  Software  configuration  audit/review  Is  evaluated  for  adherence 
to  regulations  and  standards  (such  as  MIL-STD-1S21B),  and  for  the 
planning/conduct/results  associated  with  such  audits/reviews. 

A.4  SOFTWARE  LIFE  CYCLE  PROCESS  EVALUATION  PROCEDURE. 

The  SLCP  evaluation  procedure  is  an  embedded  part  of  the  RAMSS 
and  the  general  software  supportability  evaluation  process  as  shown 
in  figure  A-2.  The  particular  SLCP  evaluation  emphasis  by  activity 
and  life  cycle  phase  applied  is  illustrated  In  figure  A-3.  The 
specific  aspects  of  the  SLCP  evaluation  are  briefly  described  In  the 
following  paragraphs. 

A.4.1  Planning  the  Evaluation. 

a.  It  is  necessary  for  the  STM/OSE  to  carefully  plan  for  the  col¬ 
lection  of  required  SLCP  data  In  order  to  adequately  complete  the 
SLCP  evaluation  questionnaire.  The  STM/DSE  should  review  the  SLCP 
questionnaire  during  AFOTEC  advanced  planning  along  with  the  likely 
sources  for  answers  tO  the  SLCP  questions.  A  timetable  should  be 
developed  as  part  of  the  evaluation  plan  which  specifies  when  the 
Identified  source  documents  will  be  available,  program  reviews  will 
be  held,  tests  will  be  conducted,  and  key  personnel  can  be  visited  to 
retrieve  the  Information  needed  to  answer  the  questions  during  the 
"official"  conduct  of  the  evaluation.  Furthermore,  problems/concerns 
noted  by  AFOTEC  personnel  during  this  planning  and  data  collection 
phase  can  be  presented  to  procurement  and  development  contractor 
personnel  for  possible  early  life  cycle  resolution. 
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Figure  A-3.  Focus  of  SLOP  Evaluation  by  Activity  and  Phase 
Applied  (Weight  Indicates  Relative  Emphasis  for 
More  Than  One  Activity  During  the  Phase) 
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b.  This  SLCP  focus  during  the  OTtE  planning  process  does  not 
cause  a  significant  change  In  the  usual  process.  Most  source 
resources,  for  question  resolution  Mill  be  similar  from  system  to 
system.  For  the  most  part,  the  SLCP  questionnaire  can  serve  as  a 
simple  checklist  for  AFOTEC  SLCP  concerns  addressed  during  OT&E  plan¬ 
ning  and  software  system  data  collection  processes. 

c.  During  the  data  collection  It  should  also  be  possible  for  the 
STM  and  OSE  to  establish  a  perspective  on  the  possible  ranges  for  the 
User/Supporter  Baseline  Estimate  data.  For  example.  Information  from 
the  development  contractor  'activity  should  Indicate  how  often  changes 
are  being  made  to  the  development  baseline  and  the  nature  of  those 
changes  (type,  complexity).  The  trend  over  time  for  these  change 
data  and  the  number  and  skill  level  of  the  development  contractor 
personnel  will  be  an  Indicator  of  the  associated  data  required  for 
the  User /Supporter  Baseline  Estimate. 

A. 4. 2  Conducting  the  Evaluation.  Conducting  the  SLCP  evaluation 
consists  of  the  formal  completion  of  the  SLCP  questionnaire  responses 
by  the  STM,  OSE  and  other  designated  personnel.  Previously  collected 
data,  updated  so  as  to  reflect  current  software  life  cycle  process 
status,  can  be  used  as  a  basis  for  the  responses.  The  User /Supporter 
Baseline  Estimate  can  be  compared  to  associated  change  data  and 
personnel  requirement  during  development  to  help  determine  the 
overall  Impact  of  the  software  maturity  upon  the  software's  support- 
ability.  Once  the  question  responses  have  been  completed,  the 
responses  are  entered  Into  the  RAMSS  automated  support  tool  data  base 
for  analysis. 

A. 4. 3  Analyzing  Evaluation  Results. 

a.  The  RAMSS  automated  support  tool  provides  SLCP  analysis 
results  In  the  form  of  evaluation  averages  at  each  level  of  the  hier¬ 
archy,  percentile  of  the  SLCP  evaluation  scores  relative  to  all 
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evaluitlon  data  base  SLCP  evaluation  scores,  and  the  relative  Impact 
of  the  SLCP  major  evaluation  factors  (project  management  and  configu¬ 
ration  management)  upon  the  overall  software  supportabllity  risk 
assessment. 


b.  The  specific  Interpretation  and  form  of  the  SLCP  analysis 
results  as  part  of  the  RAMSS  analysis  are  described  In  more  detail  In 
the  RAMSS  User's  Handbook  (reference  1.4.6). 


A.4,4  Reporting  Results  of  Evaluation.  The  SLCP  evaluation  results 
are  reported  as  part  of  the  overall  RAMSS  results.  The  form  of  these 
results  Is  dependent  upon  AFOTEC  reporting  requirements.  The  output 
from  the  RAMSS  automated  support  tool  forms  a  basis  for  the  reporting 
of  these  results  as  described  In  the  RAMSS  User's  Handbook. 


A. 5  RESPONSE  FORM. 

a.  There  Is  no  special  response  form  required  for  the  Software 
Life  Cycle  Process  evaluation.  However,  the  question  response  guide¬ 
lines  In  attachment  A1  are  organized  so  that  there  Is  one  page  for 
each  question.  Its  guidelines  for  response,  pertinent  terminology,  an 
evaluator  rating,  and  a  rationale  for  the  rating.  The  Software  Test 
Manager  (STM),  Deputy  for  Software  Evaluation  (OSE)  or  other  desig¬ 
nated  personnel  are  encouraged  to  put  a  copy  of  attachment  A1  In  a 
separate  notebook  for  each  software  system  under  evaluation  and  use 
the  question  guideline  page  as  a  permanent  record  for  comments, 
notes,  rationale,  and  the  question  score. 

b.  In  order  to  consolidate  the  Software  Life  Cycle  Process 
question  responses  for  entry  Into  the  RAMSS  data  base,  the  National 
Computer  Systems  (NCS)  answer  sheet  used  for  the  Software  Product 
evaluation  (AFOTECP  800-2,  Volume  III)  could  be  used. 
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A. 6  RESPONSE  SCALE. 

a.  To  complate  the  evaluation  questionnaire,  the  evaluator  will 
use  the  subjective  scale  of  agreement  from  1  (completely  disagree)  to 
6  (completely  agree).  In  general,  the  response  scale  should  be 
Interpreted  as  follows: 

(1)  COMPLETELY  AGREE  (6):  There  must  be  absolutely  no  doubt 

when  using  this  response  that  the  characteristic  being 
evaluated  Is  totally  satisfactory  with  respect  to  the 

characteristic  addressed. 

(2)  STRONGLY  AGREE  (S):  This  response  Indicates  that  the 

characteristic  being  evaluated  Is  very  good  and  Is  very 
helpful  for  software  supportablHty. 

(3)  GENERALLY  AGREE  (4):  This  response  Indicates  that  the 

characteristic  being  evaluated  Is  satisfactory,  but  may 
require  improvements  to  make  It  helpful  for  software 
supportablHty. 

(4)  GENERALLY  DISAGREE  (3):  This  response  Indicates  that  the 

characteristic  being  evaluated  Is  unsatisfactory,  and 

some  Improvements  are  required  to  make  It  helpful  for 

software  supportablHty. 

(5)  STRONGLY  DISAGREE  (2):  This  response  1-nd1cates  that  the 

characteristic  being  evaluated  Is  unsatisfactory  and 

major  Improvements  arc  required  before  It  would  be  help¬ 
ful  for  software  supportablHty. 

(6)  COMPLETELY  DISAGREE  (1):  There  must  be  absolutely  no 
doubt  when  using  this  response  that  the  characteristic 
being  evaluated  1s  totally  unsatisfactory  with  resoect  to 
:he  char act eristic  looressed. 
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One  of  these  responses  should  be  given  for  each  question.  Also, 
responses  1  or  6  are.  In  general,  not  expected,  since  these  responses 
Indicate  a  worst  possible  or  best  possible  characteristic  relative  to 
software  file  cycle  processes  In  general. 

b.  Note  that  the  correspondence  with  the  letters  on  the  National 
Computer  systems  (NCS)  answer  sheet  (AFOTECP  800-2,  Volume  III)  Is  as 
follows  In  case  that  answer  sheet  Is  used  to  consolidate  the  question 
responses: 

6  -  A 
5  •  B 
4  -  C 
3*0 
2  >  E 
I  •  F 

A. 7  EVALUATION  INFORMATION  SOURCES. 

The  sources  of  Information  to  determine  a  response  to  a 
question  can  be  categorized  In  many  ways.  One  convenient  categoriza¬ 
tion  Is: 

(1)  Project  Documents:  Government,  Contractor 

(2)  Regulations,  Directives,  Guidelines:  Internal,  Compli¬ 
ance 

(3)  Personnel:  Procurement,  Development  Contractor,  Opera-, 
tion  Support,  Interface  Working  Groups. 

The  primary  Information  sources  across  these  categories  are  listed  In 
figure  A-4.  The  terminology  for  some  of  the  questions  Is  based  on 
the  0o0-STD<f2167  for  Oefense  System  Software  Development, 
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Directives,  Regulations,  Standards 


1.  OoOO  5000.1,  Major  System  Acquisition,  19  Nov  1985. 

2.  DoOI  5000.2,  Major  System  Acquisition  Procedures,  19  Nov 
1985. 

3.  OoOD  5000.3,  Test  and  Evaluation,  26  Dec  1985  (DRAFT). 

4.  DoDD  5000.3  M-3,  Software  Test  and  Evaluation  Manual,  Oct 
1985. 

5.  OoOO  5000.29,  Management  of  Computer  Resources  In  Major 
Systems,  26  Apr  1976  (In  Revision). 

6.  OoOO  5000.31,  Higher  Order  Programming  Language  (HOL) 
Standardization  Policy  for  Embedded  Computers,  10  Jun 
1983; 

7.  AFR  800-14  Vol.  I,  Management  of  Computer  Resources  In 
Systems,  12  Sep  1975. 

8.  AFR  800-14.. Vol.  11,  ^Acquisition  and  Support  Procedures 
for  Computer  Resources  In  Systems,  26  Sep  1975. 

9.  AFR  55-43,  Management  of  Operational  Test  and  Evaluation, 
28  Jun  1985. 

10.  AFR  65-3,  Configuration  Management,  1  Jul  1974. 

11.  AFR  80-14,  Test  and  Evaluation,  12  Sep  1980. 

12.  AFR  800-4,  Transfer  of  Program  Management  Responsibility, 
15  Jun  1982. 

13.  AFSCP  800-48,  Software  Management  Indicators,  9*0ec  1985. 

14.  AFOTECR  55-1,  AFOTEC  Operations  Regulation,  1  Jun  1985. 

15.  OoO-STD-2167,  Defense  System  Software  Development,  4  Jun 
1985. 

16.  OoO-STO-2168,  Software  Quality  Evaluation,  24  Apr  1985 
(DRAFT). 


Figure  A-4.  Information  Sources  for  SLCP  Evaluation 
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17.  0oD-ST0-480A,  Configuritlon  Control  -  Engineering 
Changes,  Deviations  and  Waivers,  12  Apr  1978. 

18.  0O0-ST0-482A,  Configuration  Status  Accounting  Data 
Elements  and  Related  Features,  1  Apr  1974. 

19.  MIL-ST0-483A,  Configuration  Management  Practices  for 
Systems,  Equipment,  Munitions,  and  Computer  Programs, 
4  Jun  1985. 

20.  MIL-STD-490A,  Specification  Practices,  4  Jun  1985. 

21.  MIL-ST0-1521B,  Technical  Reviews  and  Audits  for  Systems, 
Equipments,  and  Computer  Software,  4  Jun  1985. 

22.  MIL-S*52799A,  Software  Quality  Assurance  Program 
Requirements,  1  Aug  1979. 

23.  Joint  Regulation,  Manaoement  of  Computer  Resources  In 
Defense  Systems,  30  Dec  1983  (DRAFT). 

24.  Joint  Regulation,  Data  Item  Descriptions,  4  Jun  1985. 

25.  TRW  Guldetfook  Series,  An  Air  Force  Guide  to  Computer 
Program  Configuration  Management,  Aug  1977. 

26.  POWER  System  Manager's  Manual,  1983. 

27.  ANSI/IEEE  Std  828-1983,.  Software  Configuration  Management 
Plans,  23  Jun  1983. 

28.  ANSI/IEEE  Std  829-1983,  Software  Test  Documentation, 
19  Aug  1983. 


Project  Specific  Documents 


1.  Program  Management  Directive  (PMO) 

2.  Program  Management  Plan  (PMP) 

3.  Test  and  Evaluation  Master  Plan  (TEMP) 

4.  Computer  Resources  Life  Cycle  Management  Plan  (CRLCMP) 


Figure  A-4.  Information  Sources  for  SLCP  Evaluation  (Continued) 
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5.  Computer  Resources  Integrated  Support  Plan  (CRISP) 

6.  Operational /Support  Configuration  Management  Procedures 
(0/S  CMP) 

7.  Development  Test  and  Evaluation  Plans 

8.  Operational  Test  and  Evaluation  Plans 

9.  Contractor  Computer  Program  Development  Plan  (CPOP) 

10.  Contractor  Software  Configuration  Management  Plan  (SCMP) 

11.  Contractor  Software  Quality  Assurance  Plan 


Figure  A-4.  Information  Sources  for  SLCP  Evaluation  (Concluded) 
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ATTACHMENT  A1 

SOFTWARE  LIFE  CYCLE  PROCESS  QUESTION 
RESPONSE  GUIDELINES 

I.  Tht  Softwart  Life  Cycle  Process  Questions  and  Response  Guide* 
lines  are  presented  In  this  attachment.  The  Information  for  each 
question  Is  presented  on  one  page  and  consists  of  a: 

(1)  Statement  of  the  evaluation  question 

(2)  Characteristic  Identification 

(3)  Applicable  actlvlty(les) 

(4)  Explanation  of  the  question  as  appropriate 

(5)  Glossary  of  terms  as  appropriate 

(6)  Special  response  instructions  (If  any) 

(7)  Response  rationale  to  be  completed  by  the  evaluator 

(8)  Response  score  to  be  completed  by  the  evaluator  (range  1s 
1  to  6). 


The  question  Identification  Information  Is  at  the  top  right  of  each 
page.  For  example,  "8CN(I0)  *  001"  1$  question  number  001  for  the 
characteristic  Identification  (ID)  within  the  major  factor  Software 
Configuration  Management  (SCM).  In  addition,  each  set  of  char* 
acterlstic  questions  (e.g.,  for  Identification)  is  preceded  by  a  one- 
page  description  of  the  characteristic  features. 


b.  The  Software  Project  Management  and  Software  Configuration 
Management  Suidelines  are  presented  in  ‘hat  order. 
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ACRONYMS 


Th«  following  list  of  acronymo  «ro  froquontly  usod  in  th 
quMtlono  and  ara  auaaarlzad  haro  for  convanlanca. 


CCB 

CDR 

CDRL 

CRISP 

CRLCHP 

CRWG 

CSC 

cscz 

OZD 

DT&B 

ECP 

FCA 

FOTAE 

FQT 

KZPO 

HOL 

HWCI 

ICWG 

Z0T6E 

ZSA 

IVCV 

JLC 

0/8  CHP 


Configuration  Changa  Board 

Critical  Daalgn  Raviaw 

Contract  Data  Raqulraaanta  List 

Computer  Raaourcaa  Zntagratad  Support  Plan 

Computer  Raaourcee  Life  Cycle  Managamant  Plan 

Computar  Raaourcaa  Working  Group 

Computer  Software  Component 

Computer  Software  Configuration  Item 

Data  Item  Oaaerlptlon 

Oavelopmant  Taat  and  Evaluation 

Engineering  Change  Propoaal 

Functional  Configuration  Audit 

Final  Operational  Teat  and  Evaluation 

Formal  Qualification  Taat 

Hierarchy*  Input*  Procaaa*  Output 

High  Order  Language 

Hardware  Configuration  Item 

Interface  Control  Working  Croup 

Initial  Operational  Teat  and  Evaluation 

Inatructlon  Sat  Architecture 

Independent  Verification  and  Validation 

Joint  Loglatlca  Commandara 

Operational/Support  Configuration  Management 
Proceduraa 


PCA 

POR 

PHD 

PMP 

RF? 

SCH 

SCN 

SDR 

SON 

SOW 

SPH 

SRR 

TEMP 

TPWG 

TRR 

WBS 


Phyalcal  Configuration  Audit 
Preliminary  Daalgn  Review 
Program  Management  Directive 
Program  Management  Plan 
Requeat  for  Quote 
Software  Configuration  Management 
Specification  Change  Notice 
Syatem  Daalgn  Review 
Statement  of  Need 
Statement  of  Work 
Software  Project  Management 
System  Ragulrementa  Review 
Teat  and  Evaluation  Heater  Plan 
Taat  Planning  Working  group 
Teat  Readlneaa  Review 
Work  Breakdown  Structure 
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SOFTWARE  PROJECT  MANAGEMENT  PLANNING 


Th«  quMtlona  SPHCPD-OOl  through  SPM<PL>-032  addrasa  adaquacy 
o£  aoftwara  projact  aanagaaant  planning  for  tha  procuraaant* 
davalopaant  contractor*  and  operation  aupport  activitiaa.  Project 
managaaant  planning  ia  aatabliahad  in  tha  form  of  technical 
documentation  that  baeomaa  more  detailed  aa  davalopmant  procaada  and 
more  rafinad  aa  tha  final  davalopmant  producta  ara  avolvad  during 
poet  daploymant  aupport.  Three  lavala  of  project  planning  ara 
generally  employed  during  tha  aoftwara  ayatam'a  life  cycle: 

(1)  Procurement  activity  project  planning 

<2>  Oavalopmant  Contractor  activity  project  planning 

(3)  Operation  Support  activity  project  planning 

Tha  Procurement  activity  project  plana  include: 

<1)  Program  Management  Plan  <PMP> 

(2)  Teat  and  Evaluation  Heater  Plan  (TEMP) 

(3)  RFP/SOW/CDRL  Package 

(4)  OTftE  Plana 

(5)  OT&E  Plana 

<6)  Computer  Reaourcea  Life  Cycle  Management  Plan  (CRLCMP) 
Computer  Reaourcea  Integrated  Support  Plan  (CRISP) 

Operational /Support  Configuration  Management 

Procedurea  (0/S  CMP) 

The  Development  Contractor  activity  project  plana  include: 

(1)  Software  Development  Plan  (8DP> 

(2)  Software  Configuration  Management  Plan  (SCMP) 

(3)  Software  Quality  Evaluation  Plan  (SQEP) 

(4)  Software  Standarda  and  Procedurea  Manual  (SSPM) 

(5)  Software  Teat  Planning 

(Plan*  Procedurea.  Oeacription.  Acceptance) 

The  Operation  Support  activity  project  plans  include: 

(1)  Computer  Reaourcea  Life  Cycle  Management  Plan  (CRLCMP) 
Computer  Reaourcea  Integrated  Support  Plan  (CRISP) 

Operational /Support  Configuration  Management 

Procedurea  (0/S  CMP) 

(2)  Software  Configuration  Management  Plan'  (SCHP) 

(3)  Software  Support  Management  Plan  (SSKP) 

(4)  Other  Agreementa  (e.g.*  Hemoranduma  of  Agreement) 

The  adequacy  of  aoftwara  project  management  planning  with 
reapect  to  the  area  of  aoftwara  aupportability  ia  mostly  a  rtatt^r  z-: 
procuramant  raquiring  aoftwara  aupportability  char  seter  is'.ica . 
davalopmant  contractor  implamanting  tha  characteriaitics.  ar.-J 
operation  aupport  tranaitloning  early  life  cycle  concepta  ar.^: 
continuing  tha  avolution  procaaa  through  tha  poat  daploymant  Iiti* 
cycle  phaaa. 
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QUESTION  DATA  SHEET 


Quest ion  Nusbsr  SPK(PL)  '  001 

OUEST^QNi  Planning  for  cosputsr  rssouress  has  bssn  adaquata  with 
rsspset~to  acquisition »  dsvslopaant*  logistics,  and  training. 

ACryVIIYlgll  Proeuranant  and  Operation  Support 

gj^y^N^TjON!  The  primary  procursaant  planning  doeuaanta  include  the 
PHP,  TEMP,  Syataa/Sagaant  Specification,  and  CRLCHP  (CRISP,  0/S  CMP) . 
The  coaputar  raaourcaa  Includaa  the  hardware,  software,  personnel, 
proeeduraa,  faellltlee,  schedule,  budget,  and  so  forth.  All  aepecta 
of  acquisition,  developaent,  logiatica  support,  and  training  must  be 
planned.  The  Hileetonea  I,  II,  and  III  provide  aajor  event  check 
points  for  analyala  and  review  of  plana.  Analyala  ahould  alwaya  be 
conducted  with  reapect  to  operational  requireaenta,  and  the  results 
integrated  back  into  the  **livin9”  plans.  Plena  for  aeaauring 
software  quality  attributes,  in  particular  software  aupportabil  ity , 
are  required. 

GLOSSARY: 


Si^\ 


RESPONSE  INSTRUCTIONS: 


RESPONSE  RATIONALE: 


5i§BSN§i.ssgRgi 

'iOCMPLITILV 


msssssa 


QUESTION  DATA  SHEET 


Qutt«tion  Number  SPiKPL)  -  0C2 

QUESTION;  Procurmment  planning  for  computer  reeourcea  hee  been 
eonaletent  with  the  eyetem  development  end  ecquleltion  plan. 

ACTIVITyc?) ;  Procurement  end  Operation  Support 

gXS^ANAT|ONj,  The  primary  ayetem  development  and  anqulaition  plan  la 
the  PMpT  The  CRLCMP  (CRISP, 0/S  CMP>  ahould  be  derived  from  the 
ellocatlon  of  the  ayetem  requirements  in  the  PHP  to  computer 
resources.  The  CRLCMP  Identifies  computer  resource  acquisition  and 
life  cycle  support  requirements.  The  CRLCMP  reflects  the  software 
development  and  support  approach  for  the  system  and  is  evolved  as  a 
living  document  throughout  the  system  life  cycle. 


GLOSSARY: 


5§§SSl!§£-li!§ISySII2!!2i 


RESPONSE  RATIONALE: 


RES£OMSE_SCgRE : _ 

“TcOMPlItELY'dIS AGREE  ■  1.2. 3. 4, 5, 6  ■  COMPLETELY  AGREE) 
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QUESTION  DATA  SHEET 


m 


QuMtion  Nuaib«r  SPfKPL)  *  003 

QUESTION;  Planning  for  coaputor  raaourcoa  has  baan  baaad  upon  an 
aequlaltion  achadula  with  adaquataly  apaclflad  allaabonaa. 

ACTIVITY  <^!)«  Proeuranant  and  Oparatlon  Support 

EXPLAN ATJOW;  Tha  plana  ahould  ba  baaad  upon  a  raaliaitlc  aequlaltion 
achadula.  Major  Hllaatonaa  1,  11,  and  III  ahould  ba  apaclflad  aa  wall 
aa  tha  major  ravlaw  and  audit  polnta  auch  aa  SOR*  SRR»  POR#  CDR»  TRR. 
FCA»  PGA.  Tha  tranaltlon  and  turnovar  dataa  ahould  alao  raallatlcally 
raflact  tha  rlaka  In  acquiring  and  supporting  auch  a  ayatan.  Studiaa 
and  analyala  ahould  hava  baan  parforaad  prior  to  tha  Full  'Scala 
Davalopmant  affort  In  ordar  to  dataralna  what  a  raallatlc  achadula 
ahould  ba. 


GLOSSARY : 


^3^ 


RESPONSE  INSTRUCTIONS; 


RESPONSE  RATIONALE; 


RESPgNSE_SCgREj_ _ 

<COMPLETELY"dIsAGREE  ■  1,2. 3, 4. 5. 6  ■  COMPLETELY  AGREE) 
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QUESTION  DATA  SHEET 


QuMtion  Number  S?H(PL)  -  004 

QUESTION ;  Coaputar  raaourcaa  hava  baan  adaquataly  addraaaad  «a  major 
eonaldaratlona  at  procuraaant  raviawa/  audita,  and  aanagamant 
avaluationa. 

ACTjyiTy<?) !  All 

HEWAMAIIQM*  Typical  procuraaant  raviawa,  audita,  and  aanat^amant 
avaluatlon  Involva  participation  of  all  actlvitlaa.  Tha  dagraa  of 
participation  dapanda  upon  tha  particular  avant.  Faaalblllty  atudiaa. 
tradaoff  analyala,  prototypa  davalopaanta ,  and  allaatona  daciaion 
polnta  dataralna  how  thoroughly  coaputar  raaourcaa  hava  baan 
addraaaad . 


GLOSSARY ! 


e 


RESPONSE  INSTRUCTIONS; 


RESPONSE  RATIONALE; 


‘(COMPLZ' 


rSLV  ^  2,  2  »  GO^IPLZTZLV  AGP.ZZ: 
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QUESTION  DATA  SHEET 


OuMtion  Nu»b«r  SPMCPL)  -  005 

QUESTION ;  Pl«nn«d  eonputsr  rmmoxircmm  hava  b««n  analyzsd  adaquataly  by 
proeur«ai«nb  to  anauro  conforaaneo  with  atatad  oparational  and  aupport 
raquir aaanta . 

Proeuraaant 

Mathoda  of  analyala  Includa  faaalbility  atudiaa, 
tradaoff  atudiaa»  rlak  analyala*  and  prototypa  davalopmant.  Thaaa 
aathoda  uaually  occur  during  Concapt  Exploration  and/or  Damonatratlon 
and  Validation  phaaaa  prior  to  tha  Full  Scala  Davalopaant  phaaa. 
During  Full  Scala  Davalopaant  an  ZV6V  function  can  aaalat  in  auch 
analyala. 


glossary; 


R^SPON^E  INSTRUCTIONS; 


RSSPQSSS-SAIISMALSi 


iCCftSj, 

” <  c5mPL2THl7“5 ISAGRii  »  1.2. 3. 4. 3. 5 
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QUESTION  DATA  SHEET 


QuMtlon  Nuab«r  SPHCPL)  -  006 

QUESTl^ONt  Proeurancnt  planning  for  softwaro  quality  attrlbutaa  haa 
baan  adaquataly  aaphaaizad  throughout  tha  aoftwara  Ufa  cycla 
aequlaltion. 

ACTj:y|Ty  <S) :  Procuraaant 

6^Pt.AMAT10W :  Softwara  quality  attrlbutaa  ahould  ba  a  na^or 
eonaldaratlon  in  tha  Initial  planning  of  aoftwara  raqulraaanta.  Thia 
aaphaala  ahould  ba  eontlnuad  throughout  tha  ayataa  and  aoftwara  Ufa 
cycla  phaaaa.  Softwara  quality  raqulraaanta  and  raaponalbiiltiaa 
ahould  ba  daflnad  in  tha  CRLCHP  <CRISP»  0/8  CMP) .  Procaduraa  should 
ba  davalopad  and  laplanantad  to  anaura  propar  aaaaaanant  of  eoaputar 
raaourcaa  throughout  tha  ayataa  Ufa  cycla.  Tha  procuraaant  activity 
ahould  davalop  asaaaaaant  procaduraa  to  anaura  that  tha  eoaputar 
aoftwara  will  aaat  aanagaaant  pollclaa  and  appropriata  ragulations. 
confora  to  atandarda#  and  aaat  parforaanca  and  quality  raqulraaanta 
throughout  tha  ayataa  Ufa  cycla.  Coaputar  aoftwara  ahould  ba 
contlnuoualy  aaaaaaad  by  aaana  of  ravlawa»  audita,  varlfication 
validation  taatlng,  and  othar  anforeaaant  aetivltlas. 

G^SSARYi 

Sfiiisfss _ _ AiiliPyiSSi  Rallablllty,  Supportabillty, 

Maturity,  Efflelaney,  and  ao  forth. 


RESPgNSE_INSTRyCTigNS: 


RESPONSE  RATIONALE: 


# 


’•.CCMPLiTiL7"5IiAGREH”«~’l  .2.;3.4.3.!9  »  OCJIPLITELY  AGSr-E 
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QUESTION  DATA  SHEET 


QuMtion  Number  SPHCPL)  *  007 


SUESIIQiil  Herglns  for  reserve  computer  resource  capacity  to  provide 
for  later  product  iaprovements  are  adequate. 


ACTIVITYCS):  All 


:  Requireaenta  for  aarglne  and  the  initial  values  are 
eebabliahed  by  the  procurement  and  operation  support  activity  in  the 
PNO#  RFP/SOW,  and  Systea/ Segment  Specification.  These  margins  ere 
then  evolved  throughout  the  Full  Scele  Developaent  by  the  developaent 
contractor  activity.  Margins  should  be  established  for  meaory. 
external  storage,  task  utilization,  terminal  usage,  performance 
parameters,  and  so  forth.  Typical  guidelinaa  are  to  leave  30  to  50 
percent  of  the  total  resource  capacity  as  reserve  dependent  upon  the 
resource  and  the  particular  application.  The  margin  of  reserve  is 
very  important  for  software  supiport ability  since  changes  will  usuail'/ 
require  consumption  of  some  of  the  reserve. 


Gl^SSAR]^ 

The  difference  between  the  total  available  capacity  of 
resource  and  the  actual  amount  used. 


RESPgMSE_INSTgUCTIQM§i 

"  F7IT~S0m'‘or~’leaa  of  required  reserve  capacity  is  available  for 
at  least  one  of  the  resources,  or  no  margin  requirements 
have  been  established  , 

E/2:  90Z  to  SOX  of  required  reserve  capacity  is  aveilable  for 
at  least  one  of  the  resources 
D/3:  SOX  to  70X 
C/4:  70X  to  80X 
3/9:  sox  to  90X 

A/ IS  90X  to  lOOx  of  required  reserve  capacity  is  available  for 
all  resources 


RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

“ CcSfiPLETiLY'SISAGRii' 


1,2. 3, 4, 9, 6 


COMPLETELY  AGREE) 
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QUESTION  DATA  SHEET 


QuMtion  NuMbar  SPH(PL)  -  008 

OUESjTIQN  AccaptablA  tachniqu^a  hava  baan  uaad  to  aatlaata  and 
monitor  aoftwara  coata  throughout  tha  ayatam  Ufa  cycla. 

ACTIVITY <S) ;  All 

nacaaaary  that  a  standard  tachnlqua  <a.g.,COCOHO 
modal)  ba  uaad  to  aatlaata  aoftwara  coata  throughout  tha  ayatam  Ufa 
cycla.  It  la  probably  mora  important  that  ftfimf  tachnlqua  ba 
conaiatantly  uaad  and  tha  raaulta  carafully  nonitorad  than  which 
tachnlqua  is  uaad.  Each  activity  should  hava  aona  mathod  which  la 
uaad  and  a  way  to  corralata  coat  raaulta  from  tha  othar  actlvitlaa 
with  thair  raaulta.  A  coat/achadula  riak  analyala  should  ba  dona  at 
aach  of  tha  major  projact  Ufa  cycla  daclaion  points.  Tha  aoftwara 
coata  ara  usually  ralatad  to  tha  ralatad  WBS  tasks  in  ordar  to 
accomplish  tha  coat/achadula  riak  analysis. 

CLggSARYr 

Conatruetiva  Coat  Modal  (B.  Boahm  of  TRW) 

raaourcaa  conaumad  to  davalop  and  support 
aoftwara  throughout  its  Ufa  cycla. 


RESPONSE  instructions: 


RESPONSE  RATIONALE: 


RESPONSE  SCORE: 
”cC0MPLETELY"5iSAGRii 


1.2. 3. 4. 3,6 


COMPLETELY  AGREE) 


;aT.«  JHEE? 


Al-11 


UBManiooQoa 


Ag.  'kg. 

•a*"  ti*  I  • 


QuMtion  Nuabar  SPH(PL>  - 

QUESTION!  Tha  CRLCHP  (CRISP,  0/S  CMP)  eontalna  adaquata 
apacif  icatlona  of  tha  acquialtion  raquiraaanta  for  computar 
raaourcaa . 

Procuraaant  and  Oparatlon  Support 

During  tha  davalopaant  phaaa,  tha  CRLCHP  (CRISP, 0/S  CMP) 
aarvaa  to  daflna  tha  davalopaant  plan  and  to  idantlfy  tha  computar 
raaourcaa  naeaaaary  to  aupport  tha  ayataa  aftar  daployaant.  Aftar  tha 
ayataa  haa  baan  daployad,  tha  CRLCHP  aarvaa  to  daflna  tha  plan  for 
managing  tha  aupport  of  tha  ayataa' a  computar  raaourcaa.  Tha 
procuramant  activity  ahould  bagln  praparlng  tha  CRLCHP  with  tha  halp 
of  tha  oparatlon  aupport  activity,  with  complatlon  no  latar  than 
Hllaatona  II  or  aqulvalant.  At  thla  point  tha  CRLCHP  ahould  focua  on 
plana  for  davaloplng  tha  computar  raaourcaa.  Including  computar 
raaourcaa  naadad  for  ayataa  aupport.  Aa  tha  ayataa  prograaaaa  into 
tha  Full  Scala  Davalopaant  phaaa,  tha  CRLCMP  ahould  ba  axpandad  tc 
provida  a  comprahanalva  plan  for  aupport  of  computar  raaourcaa.  3y 
Hllaatona  III  or  aqulvalant,  tha  CRLCMP  ahould  contain  a  plan  for 
tranaltlonlng  computar  raaourca  raaponalbllltlaa  from  tha  procuramant 
activity  to  tha  oparatlon  aupport  activity. 

(?^OSSARY ! 

m 


RgSPONSE^INSTRUCTIONSi 


RESPONSE  RATIONALE: 


S§5S9!!5i_5S95§i 

” (COMPLiTELY^DISAGRii' 


1,2, 3. 4, 5, 6 


COMPLETELY  AGREE) 


Al-12 


QUESTION  DATA  SHEET 


QuMtlon  Number  SPH(PL>  ~  010 

SilSSIIQlil  CRLCMP  (CRISP*  0/S  CMP)  edequately  addresaee  the 

rMponclbilltiM  and  prodedurea  to  anaura  proper  software 
configuration  aanagamant  throughout  the  system  life  cycle. 

Procurement  end  Operation  Support 

Responsibilities  should  be  defined  in  the  CRLCMP  (CRISP, 
0/S  CMP)  and  proesduras  should  be  developed  and  implemented  to  ensure 
proper  control  of  computer  resources  throughout  the  system  life  cycle 
in  accordance  with  AFR  6S>3.  Computer  hardware  and  software  should  be 
identified*  specified,  and  managed  aa  configuration  items.  The 
mechanism  for  controlling  computer  hardware  and  software  changes  is 
the  documentation  for  each  configuration  item,  and  it  is  the 
raaponaibility  of  the  system  configuration  manager  within  the 
procurement  or  operation  support  activity  to  ensure  that  this 
documentation  is  accursts  and  current.  The  Configuration  Control 
Board  (CCB)  should  be  the  primary  authority  for  approving  hardware 
and  software  changes  throughout  the  system  life  cycle.  Membership  on 
the  CCB  should  be  datarmined  by  the  procurement  and/or  the  operation 
support  activity  in  accordance  with  the  provisions  of  AFR  65*3. 

GLOSSARY  t 


RESPONSE  instructions: 


RESPONSE  RATIONALE: 


# 


RESPONSE  SCORE: 

(^MPLETELY~5lSAGRii  «  1,2, 3*4, 5, 6  >  COMPLETELY  AGREE) 


Al-13 


QUESTION  DATA  SHEET 


QuMtlon  Nu«b«r  SPKCPL) 


fi!2ESIIQNl  Th«  projact  managancnt  rasponalblllty  for  intagrating 
coaputar  raaourcaa  into  a  ayataa  haa  raaainad  cantralizad  throughout 
tha  lifa  of  tha  ayataa. 

ACTjy.|TY<S2i  All 

fXgl^^MAXION ;  Tha  ainiaua  raquiraaant  ia  that  aach  aetivity'a  pro^act 
aMagMant  raaponaibility  raaain  with  tha  aaaa  organizational 
alaaant.  For  axaapla*  tha  iaplaaantlng  coaaand,  davalopaant 
contractor#  uaing  coaaand#  and  aupporting  coaaand  raaain  tha  aaaa.  A 
aora  iaportant  aapact  of  tha  cantralization  ia  that  tha  lowar  laval 
organization  atructura  < Including  paraonnal)  within  aach  activity 
ahould  raaain  intact  without  fragaantation  or  aajor  varlanca  of 
raaponaibility  ovar  all  lifa  cycla  phaaaa. 


GLOSSARY; 

Locatad  within  tha  aaaa  organizational  alaaant. 


RESPOMSE^INSTRyCTIONSi 

F/l:  Oa  to  sox  of  tha  pro^act  aanagaaant  acroaa  all 

activitiaa  and  all  phaaaa  haa  raaainad  cantralizad 
E/2:  sox  to  60X 
D/3;  GOX  to  70X 
C/4;  70X  to  80X 
B/S:  BOX  to  sox 
A/6;  sox  to  lOOX 

RESPONSE  RATIONALE; 


RESPONSE  SCORE;  _ _ _ 

"cCCMPLiTiilv’SlSAGni-  «  1.2. 3. ^.3, 5  »  :C;iPLlTZL7 


Al>14 


QUESTION  DATA  SHEET 


QuMtlon  Nuab«r  SPH(PL)  -  012 

QUESTION;  Th«  CRWC  organization  haa  baan  adaquata  throughout  tha 
ayataa  Ilf a  cycla. 

ASI2yiHi82JL  Procuraaant  and  Oparation  Support. 

S<£lei!il!/^I29!!l  alraady  aatabliahad  during  tha  Concapt 

Escploration*'  phaaa,  tha  CRW6  ahould  ba  foraad  aarly  In  tha 
Daaonatratlon  and  Validation  phaaa  to  aaalat  tha  program  nanagar  in 
planning  and  Implaaantlng  aoftwara  laauaa,  actlvltlaa,  and  functlona. 
During  aequiaitlon*  tha  laplanantlng  command  chalra  and  managaa  tha 
CRWC.  Tha  CRWG  ahould  alao  Ineluda  tha  ualng  command,  aupportlng 
command,  and  parhapa  rapraaantatlvaa  of  othar  organlzatlona 
raaponalbla  for  DT&E,  OTAE  and  training. 


GLOSSARY; 


RESPONSE  instructions; 


RESPONSE  rationale; 


response  SCORE!  _ 

~7cOMPLETELY~dIs agree  >  1,2,3,4,S,6  >  COMPLETELY  AGREE) 


Al-15 


QUESTION  DATA  SHEET 


.QuMtlon  Number  SAM  (PL)  -  013 

OUESTIQN  t  The  CRWG  hee  had  clearly  specif  led  reeponelbilltlee  end 
appropriate  authority  to  Isplesent  those  responsibilities  throughout 
the  system  life  cycle. 

dSIiyilXiSlJ.  Procurement  end  Operation  Support 

gXP^yATIQN^  The  CRWG  supports  the  program  manager  In  developing  the 
CRLCMP  TcRISP)  .  Also#  the  CRWG  recommends  sltemstlvea  In  areas  such 
as  documentation  requlremants#  software  security  requirements#  IV&V# 
standard  equipment#  standard  HOLs#  standard  ZSAs#  and  margins  for 
reserve  computer  capacity.  The  CRWG  Identifies  the  computer  resources 
end  facilities  required  to  support  the  system  throughout  the  system 
Ufa  cycle.  For  programs  with  intersarviclng  potential,  the  CRWG 
includes  members  from  each  organization  affected  by  Intersarviclng. 
The  CRWG  analyzaa  Intaraarvicing  potential  to  support  the  program 
manager's  decision  concerning  a  joint  service  facility.  This  analysis 
should  consider  operational  needs#  Ufa  cycle  costs#  technical 
capability#  and  service-unique  standards  for  computer  resources. 
Without  the  proper  authority  to  Implement  Its  decisions  and  have  its 
recommendstlons  acted  upon#  the  (HtWG  will  be  deficient. 

SkSSSARY:  A 


RESPONSE  instructions: 


RESPONSE  RATIONALE! 


■•^EjPQNSS  3CwR£:  _ _ 

(C0MPL£7iLY“5ISAGRii~”L.2.3,4.;.S  ^  C0MPL2TEL?  AGREE) 


QUESTION  DATA  SHEET 


QuMtion  Number  SPH(PL)  -  014 

QUESTION;  Thtt  CRWG  properly  asaurad  that  coaputar  raaourcaa 

comply  with  aatabllahad  policy#  procaduraa#  plana#  and  atandarda. 

ACTIVITYCS) ;  Procuramant  and  Operation  Support 

JIQN ;  Tha  CRWG  aaaiata  in  anauring  that  coaputar  raaourcaa 

comply  with~’aatabliahad  policy#  proeaduraa#  plana#  and  atandarda.  Tha 
CRWG  continuoualy  aupporta  tha  procuramant  activity  in  planning  tha 
coaputar  raaourca  lifa  cycle  and  evaluating  developed  computer 

reaourcea.  The  CRWG  recoaaenda  updatea  to  the  CRLCHP  (CRISP)  to 
enaure  that  acquiaition#  uaer#  and  aupport  raquireaenta  are 
aatiafied.  The  CRWG  evaluatea  coaputar  aoftware  plana#  producta#  and 
propoaad  changea  to  enaure  compatibility  with  tha  CRLCMP  (CRISP)  and 
conaiatancy  with  policiea  and  proeaduraa.  The  CRWG  alao  aupporta  the 
procuramant  activity  in  tha  raaolution  of  iaauea  auch  aa 

documentation  raquireaenta  and  aupport  agreementa.  If  computer 
raaourcaa  davelopaant  ia  part  of  an  intaraerviee  program#  then  the 
interaerviee  CRWG  verifiea  that  the  required  computer  reaourcea  and 
operatioti  aupport  eapabilitiea  are  available  to  aupport  the  ayaten. 
lefore  Hilaatone  III  or  equivalent#  the  CRWG  .ahould  develop 
interaerviee  proeedurea  for  operation  aupport  of  tha  ayatem.  In 
addition#  tha  CRWG  enaurea  that  the  joint  aervice  operation  aupport 
activity  partieipatea  in  the  Full  Scale  Development  phaae#  thereby- 
acquiring  tha  neeaaaary  familiarity  and  axpariance  to  aupport  the 
ayatam  upon  comp lat ion  of  davelopmant. 

GLOSSARY! 

RESPONSE  INSTRUCTIONS: 


REgPgNS|_RATigNAtii 


RESPONSE  SCORE!  _ 

”<C0MPLiTiLY“5lSAGREE  »  1#2#3#4#5#6  «  COMPLETELY  AGREE) 


Al-17 


QUESTION  DATA  SHEET 


QuMtlon  Nuabar  SPH<PL)  ~  015 

QUESTION:  Software  quality  aaaaaaaant  procaduraa  hava  baan  adaquataly 
dafinad  to  aaat  aanagaaant  policlaa  and  appropriata  ragulatlona# 
conform  to  atandarda,  and  aaat  parforaanca  and  quality  raqulramanta 
throughout  tha  ayataa  Ufa  cyela. 

ACTIVITYCS);  All 

Softwara  quality  raqulramanta  and  raaponalbllltl^s  ahall 
ba  daflnad  In  tha  CRLCMP  (CRISP).  Procaduraa  ahould  ba  davalopad  and 
laplaaantad  to  anaura  propar  aaaaaaaant  of  coaputar  raaourcaa 
throughout  tha  ayataa  Ufa  cycla.  Tha  procuraaant  activity  davalpa 
aaaaaaaant  procaduraa  to  anaura  that  tha  coaputar  aoftwara  will  aaat 
managaaant  policlaa  and  appropraata  ragulatlona.  conform  to 
atandarda.  and  aaat  parforaanca  and  quality  raqulramanta  throughout 
tha  ayataa  Ufa  cycla.  Coaputar  aoftwara  ahould  ba  continuously 
aaaaaaad  by  aaana  of  ravlawa*  audita^  varlflcatlon  validation 
taatlng,  and  othar  anforcaaant  actlvltlaa.  Tha  primary  softwara 
quality  ragulatlona/atandarda  ara  HIL*S-S2799A  and  tha  aora  racant 
Do0-ST0>216A. 


GLOSSARY! 


RESPONSE ,,  INSTRUCTIONS: 


RESPONSE  RATIONALE: 


RESPONSE^SCOREj. _ 

“<C0MPLETiLY"5iSAGRii  ■  1.2, 3, 4, 9, 6  ■  COMPLETELY  AGREE) 


QUESTION  DATA  SHEET 


Qu«stlon  Number  S?H(PL)  -  016 

QUESTION;  Planning  for  DT6E  of  eonputar  rasourcaa  haa  baan  adequate 
throughout  the  ayatan  life  cycle. 

ACTIVITY (S) !  ■  Procurement 

gXPl»AHAT^9^ i  The  primary  high  level  planning  document  for  DT&E  la  the 
TEMP.  DT&E  deacrlptlona  In  the  TEHP  ahould  concentrate  on  technical 
goala*  threaholda*  and  obsectlvea.  At  each  review  phaae.  the 
aaaantlal  queatlona  ahould  continue  to  be  whether  objectlvea  were 
met,  degree  of  confidence  In  reaulta,  and  apeclflc  ayatem  behaviour 
leading  to  obaerved  anoaallea.  The  detailed  DT&E  plana  aupplement  the 
TEMP  and  provide  Inalght  Into  the  apeclflc  aoftware  and  ayatem 
Integration  teata  and  procedurea  which  are  planned. 


GLOSSARY; 


RESPONSE  instructions: 


RESPONSE  RATIONALE; 


RESPONSE  SCORE;  _ 

”7c5MPLETiLY“5iSAGREE  ■  1,2, 3, 4, 5, 6  ■  COMPLETELY  .  AGREE) 


Al-19 


3 


I'Soail 


QUESTION  DATA  SHEET 


QuMtlon  Nuabsr  SPM<PL>  *■  017 


QUESTION,;  Planning  for  0T6E  of  conputar  raaourcas  has  baan  adaquaba 
throughout  tha  ayataa  Ufa  cycla. 

ACTIVITYCS) ;  Proeuraaant 

gXPLANATIQNl  Tha  priaary  high  laval  planning  docuaant  for  OT&E  la  tha 
TEMP.  Mora  datallad  0T6E  plana  supplaaant  tha  TEMP.  Tha  typas  of  0T6S 
ara  lOT&E  and  FOTfrE.  I0T6E  la  tha  first  taat  of  a  eoaplata  ayataa  and 
support  alaaanta  In  an  oparatlonal  anvlronaant.  I0T6E  provldaa  an 
aarly  ovar-tha>ahouldar  affort  during  tha  Daaonatratlon  and 
Validation  phaaa  of  tha  ayataa  Ufa  cycla.  Tha  purpoaa  of  tha  aarly 
IOTAS  la  to  provide  an  oparatlonal  Input  to  tha  Initial  davalopaant 
prograa,  aaaura  tha  coupling  of  raqulraaanta  to  tha  davalopaant 
prograa*  davalop  an  Intarfaca  batwaan  davalopar  and  uaar,  and  rafina 
oparatlonal  laauaa  for  aubaaquant  taat  and  evaluation.  After  go-ahaad 
for  Full  Seale  Davalopaant*  a  later  I0T6E  la  conducted  prior  to  tha 
production  daelalon.  FOT&E  la  conducted  whan  tha  ayataa  la  fully 
deployed  In  order  to  aaaaaa  full  ayataa  capability.  Specific 
objaetlvaa*  aubob^aetlvaa*  and  aaaauraa  of  affactlvanaaa  should  be 
addraaaad  In  tha  OT&E  planning  doeuaanta. 


g^OggARY: 


REgPgNSE^INSTRUCTigNSi 


55S5Q!!§5_BATigNALEi 


”7cGMPL2TiLY“5lSAGRii'“»”l.2,3,4,3.S  »  CCMPLHTEL7  AGREE) 


QUESTION  DATA  SHEET 


QuMtlqn  Nu«b«r  SPHCPL)  -  018 

QUESTION;  Software  standards  hava  baan  adaquataly  apaciflad 
throughout  tha  softwara  Ufa  eycla. 

ACTIVITY<S);  All 

RFP/CDRL/SOW  raflaet  raqulrad  softwara  standards  as 
apaciflad  by  tha  proeuraaant  activity.  Tha  standards  should  apply  to 
all  aapaeta  of  tha  softwara  davalopaant  and  support.  Typical 
standards  ara  0oD-STD-21S7»  Do00-STD-216dr  HIL-STD-483A. 

HIL-STD-1S21B,  DoO-STO-lBlSA,  and  various  ANSI/IEEE  softwara 
anginaaring  standards.  Tha  davalopaant  contractor  must  comply  with 
tha  proeuraaant  raquiraaanta  through  intarnal  standards  and 
convantiona.  Tha  oparation  support  activity  contributaa  to  tha 
racoaaandationa  on  tha  uaa  of  raqulrad  standards  and  aats  intarnal 
standards  and  convantiona  for  uaa  during  tha  post  daploymant  support 
of  tha  softwara. 

GtOSSARY^ 


RESPgNSE_IN§TRUCTigN5i 


5§55S!J5I_SAII91!iilsSi 


(.0CMPL2THL :  JI3AGPEE 


»  1.2, 3. ‘*.5,0  «  •:ChPLaTHLY  AGREE) 


QUESTION  DATA  SHEET 


# 

QuM^lon  Number  SPH(PL)  **  019 

. 

QUESTION I  Th«  planning  for  organic  and/or  contractor  aupport  during 
post  daployaant  aoftwara  support  has  baan  adequate . 

ACTIVIT^ig) 1  Operation  Support 

Software  aupportablllty  characteristics  Include  software 
product  salntalnablllty#  software  support  resources »  and  aoftwara 
life  cycle  r^ocesses.  Plans  for  orgsnlc  snd/or  contractor  aupport  and 
the  evaluation  of  whether  such  support  will  be  adequate  relative  to 
software  aupportablllty  characterlstlca  should  be  contained  In  the  | 
CRLCMP  (CRISP,  0/S  CMP). 


GLOSSARY! 


R^^PQWSE . INSTRUCTIONS ! 

s 


91 

L 

RESPOMSE  RATIONALE:  I 


RESPONSE  SCORE: 

(C0HPLETELY*5zSAGRii~«  1,2, 3. 4, 5, 6  «  COMPLETELY  AGREE) 


Al-22 


QUESTION  DATA  SHEET 


QuMtion  Numbar  SPH(PL)  ~  020 

fiU&SXIQlil  Contractual  docunanta  hava  axpilcltly  aatabliahad 
Govarnaant  ri9hta  to  ail  coaputar  raaourcaa  raqulrad  to  davalop* 
oparata*  aiaulata*  taat»  and  aupport  tha  aoftwara. 

6SXiyUXi92I  Procuraaant  and  Oparatlon  Support 

Contractual  docuaanta  ahould  claarly  aatabliah 
govarnaant  righta  to  all  coaputar  raaourcaa  raqulrad  to  aupport  tha 
aoftwara.  Tha  rlghta  aay  hava  proprlatary  elauaaa  which  auat  ba 
earafully  undaratood  by  all  partiaa.  Tha  application  aoftwara*  ayatam 
aoftwara*  and  taat  aoftwara  ahould  ba  conaldarad. 


SkSSSARYj, 


Rg§PONSg^lN5TRySTigM5i 


BS9£Sl!SS.BATZ0NAiggi 


RESPONSE  SCORE: _ 

“(COMPLETELY'DiSACRiE  ■  1*2*3*4*9*6  ■  COMPLETELY  AGREE) 


QUESTION  DATA  SHEET 


QuMtlon  Nuab«r  SPNCPL)  -  021 

QUESTION ;  Planning  for  risk  analyaia  to  Idahtlfy  araaa  of  conputar 
raaeurea  riak  haa  baan  adaquata. 

ACTIVIXYiSlt  Procuraaant  and  Oparation  Support 

*  A  eoaaon  aourea  of  oparational  aoftwara  problana  ia  tha 
difficulty  of  maintaining  end  supporting  tha  aortwara  onea  it  ia 
daployad.  Tha  tachnology  uaad  to  daaign  and  implamant  tha  aoftwara 
may  significantly  affact  this  ability.  Oaqgar  signals  may  includa  tha 
usa  of  propriatary  toola  and  tachniquaa  that  will  not  ba  availabla  to 
anginaars  aftar  ayatam  dalivary.  Altarnativaly,  thara  may  ba  uniqua 
aspacts  of  tha  daaign  affort  that  poaltivaly  affact  subaaquant  lifa 
cyCla  cost  and  affort.  Ona  approach  to  raducing  long-tarm  lifa  cycia 
riaks  is  to  anforca  tha  usa  of  common  tachnology  throughout  tha 
davalopmant  and  opa:^*ation  of  tha  aoftwara.  It  ia  not  uncommon  for  thii 
pro^act  offica  to  aupply  tools  and  aupport  aoftwara  to  tha 
davalopmant  contractor  to  ansura  commonality.  Howavar^  cara  should  ba 
axarcisad  to  avoid  Govarnmant  liability  in  casam  of  inadaquata 
Covarnmant  furniahad  tools.  Idaslly*  lifa  eycla  charactariatica  of 
oparational  significanca  should  ba  listad  aa  raquirad  charactariatica 
of  tha  systam  and  taata  should  ba  plannad  to  addrass  tha  iaauaa  that 
arisa  from  thasa  charactariatica. 

C^gSSAjRYi 


RESPONSE  instructions: 


RESPONSE  RATIONALE: 


SSSSQNSS-SSSSSi  _ 

CCMPlItSL?"  2 13AGSEE 


4.3,3  * 


QUESTION  DATA  SHEET 


QuMtlon  Nuabar  SP2(<PL>  -  022 

QUESTION ;  A  iiiaaion/funct.lon  matrix  (or  aqulvalant>  claarly 
Idantlflaa  primary  functional  eapabilitlaa  to  ba  iaplanantad  by  tha 
aof twara . 

ftS7iyiX]fiS2l  Procuramant  and  Oparatlonal  Support 

A  miaaion/f unction  matrix  (or  aquivalant  narrative)  ia 
tha  primary  aourca  of  information  about  how  tha  ayatan  capabilitiaa 
hava  baan  partitioned  between  hardware  and  aof  twara.  Thaaa  partitiona 
will  ba  important  in  determining  required  charactariatica*  in 
defining  arror/failura  criteria*  and  in  iaolating  and  correcting 
daficianciaa  noted  during  taating.  Therefore*  it  may  ba  important  to 
datarmina  that  proper  engineering  atudiaa  have  lad  to  tha 
aatabliahmant  of  thaaa  partitiona.  An  undaratanding  of  tha  aourcaa  of 
riak  in  each  of  tha  aoftwara*implamantad  functiona  identified  in  tha 
miaaion/f unction  matrix  ia  an  aaaantial  part  of  tha  overall  riak 
aaaaaaaant. 

G^0§SARYi 

SlfSi9S2£y&SlriSS...liS£XilL&  Corraapondanca  relating  primary 
functional  capabilitiaa  that  muat  ba  daaonatratad  by  taating  to  tha 
miaaion  to  ba  performed  and  tha  concept  of  operation. 


RESPONSE  INSTRUCTIONS: 


RESPONSE  RATIONALE: 


RESPONSE  jCCRE; 
““cOWPLiTHLY'DISAGRii' 


»  1.2.3.4.5.S  «  COMPLSTHLY  AGREE) 


A1-2S 


QUESTION  DATA  SHEET 


QuMtion  Nuab«r  SPM(PL>  -  023 

fiUESXIQHl  Planning  for  intoroporabllity  with  othor  aystann  haa  baan 
adaquataly  addraaaad. 

dSJ2y.2J2i92J.  Proeuraaant  and  Oparablon  Support 

S(£lr£lf^ZlSlLl  Whan  raquirad,  ayatana  auat  ba  intaroparabla  with  othar 
ayataaa  aaployad  by  tha  U.S.  and  Alllad  military  forcaa. 
Zntaroparablllty  raqulrananta  ahould  ba  idantlfiad*  daflnad, 
valldatad*  and  Ineludad  in  approprlata  planning  documantatlon  prior 
to  tha  and  of  tha  Oamonatratlon  and  Validation  phaaa.  Both 
davalopaant.  and  poat  daploymant  aupport  conearna  ahould  ba  addraaaad^ 
and  organizational  raaponaibilitiaa  dafinad. 


G^gSSARYi 


RESPgNSE_INST5yCTigNS2 

A/6:  No  intaroparability  raqulramanta  axiat  for  tha 
aubjact  ayatam 


5£9£Sif9S.5ATigNAtEj, 


RESPgNSE_SCORE: 

“<c5MPLETiLY’0ISAGRii  ■  1.2, 3, 4, 9, 6  ■  COMPLETELY  AGREE) 


Al-26 


QUESTION  DATA  SHEET 


Quest Ion  Nusbsr  SPMCPL)  -  024 

QUESTION;  Prior  to  ssch  system  milestone,  Intsrservlelng  potsntlsl 
end  life  cycle  cost  implicstlone  of  softwere  support  options,  hsve 
been  spproprletsly  Addressed. 

Procurement  end  Operstlon  Support 

gyp^/lHATIOH ;  Before  esch  system  mllsetone,  Intsrservicing 
should  be  reviewed  end  the  msnsgement  end  life  cycle  cost 
implicstions  of  msjor  softwere  support  options  should  be  snslyzsd. 
This  snelysls  should  slso  consider  Impsct  on  opsrstlonsl  needs, 
conflgurstlon  msnsgement,  end  system  Integrstlon.  For  interssrvice 
systems,  the  CRWG  should  be  en  Intsrservlcs  CRWG  that  includes 
rspresentstives  from  mil  cognizant  organizational  elements.  The  CRUG 
Clnterservlce  or  single  service)  should  ensure  that  analysis  is 
performed  to  determine  the  optimum  support  approach,  document  this 
analysis,  and  make  recommendations  to  the  procurement  activity 
concerning  the  support  approach. 

GLOSSARY : 


RESPONSE  INSTRUCTIONS; 


RESPONSE  RATIONALE: 


RESPONSS^SCOREi. 

"7cOMPLiTiLY'’5lSAGRii  ~i,2,3,4,5,6  -  COMPLETELY  AGREE) 


Al-27 


QUESTION  DATA  SHEET 


QuMtion  Nuabcr  SPM<PL>  -  02S 

QUESTION:  Th*  procur«a«nt  «nd  oparatlon  support  planning  docuaants 
hava  baan  adaquat:aly  updatad  aa  living  docuaanta  thrdughou't  tha 
aystan  lifa  cycla. 

fiSIiyilliSIl  Procuraaant  and  Oparation  Support 

EjffVAyATIOl?  1  oi  tha  docuaanta  which  should  ba  continually 
updatad  ineluda  tha  TEMP*  CRLCMP  <CItXSP»  0/S  CMP>«  and  tha  0T6E  and 
0T6E  plana.  It  ia  important  that  thaaa  docuaanta  ara  working  plans 
and  aa  auch  track  cloaaly  tha  prograaa  and  currant  atatua  of  tha 
ayataa . 


G^SSARYJ, 


RESPONSE  instructions: 


RESPONSE  RATIONALE: 


SSSBSSSE.SCOREi  _ 

CCMPLlTiLY'uTSAGiis  » 


AGS  EE  V 


QUESTION  DATA  SHEET 


OuM^lon  Numb«r  SPH(PL>  -  026 

question;  Th«  prlnelplM  «nd  ■•thodologi«s  provided  in  the 
reguletlona  heve  been  epproprietely  ineorpereted  into  the  softwere 
teet  end  eveluetion  plena. 

ASTiyiI2i5>i  All 

Typical  procurement  end  operation  support  regulations 
include  DoO  S000.3,  DoDD  S000.29,  DoDD  9000.31,  APR  600-14  (Vola  I 
and  IZ>,  APR  60-14,  and  APR  99-43.  Typical  development  contractor 
compliance  regulations  include  OoO-STD-2167,  OoO-STD-216d, 
HIL-8TD-1921B,  HZL-S-92799A  (to  be  superceded  by  OoD-STD-2168) .  The 
software  teat  and  evaluation  plana  are  primarily  contained  in  the 
TEMP,  DT6E  plana,  0T6E  plana,  IVAV  plana,  end  development  contractor 
plena . 


GLOSSARY; 


RESPONSE  INSTRUCTIONS; 


RESPONSE  rationale; 


RS2£QNSS.SSQ5Si  _ 

CChPLZTSLV  DISAGREE  = 


Al-29 


QUESTION  DATA  SHEET 


QuM^ion  Nuab«r  SPM(PL)  -  027 

OUESTIOH;  Plenning  for  ay«tM«tic,  quantitativo,  and  objactlvaly 
raportabla  aoftwara  baata  haa  baan  adaquaba. 

ACTIVITY <S) :  ALL 

ahould  ba  apparanb  from  bha  daacripbion  of  bha 
aofbwara  baaba  eonducbad  and  bhair  raaulba  whabhar  or  nob  pravioua 
goala  hava  baan  nab  and  baab  objaeblvaa  hava  baan  aabiaflad.  Vagua 
rafarancaa  bo  '*aueeaaaful  aofbwara  raaulba**  or  '*no  problana  wibh  bha 
aofbwara**  ahould  nob  ba  accapbabla.  In  ordar  bo  avaluaba  bha  prograaa 
of  aofbwara  baablng  bo  daba#  bhara  auab  ba  axpllelt  rafaranca  to:  1) 
a  ayabanable*  aeianbif leally  aound  approach  bo  carrying  oub  bha  baab, 
2)  bha  ralabionahip  babwaan  bha  ayabaaablc  baab  approach  and  bha  baab 
objacblvaa  for  bha  currant  phaaa,  3>  tha  raaulta  of  tha  taat,  and  4> 
bha  plana  for  raaolubion  of  arrora. 

glossary; 


RESPON$g_RATIQWA^g£ 


5§§EQl!§i-SC0REi 

”(c5MPLETiLY“5iSAGRii”«”l,2,3,4,5,6  ■  COMPLETELY  AGREE) 


question  data  sheet 


QuMtlon  Number  SPH(PL)  -  028 

QUESTION Planning  for  aharing  of  aoftwara  taat  rasulta  acroaa 
llfacycia  phaaaa  and  among  taat  organizatlona  haa  baan  adaquata. 

ACTIVITY<S) ;  All 

DoDD  5000.3  raquiraa  Involvaaant  and  data  aharing  acroaa 
•11  Ufa  cycl#  phaaaa  for  taat  organizatlona:  0T6E,  OTAE*  IV6V  and 
tha  davalopaant  contractor.  Tha  TEMP  la  tha  primary  high  laval 
planning  doeuaant  for  aoftwara  taat  and  avaluatlon  with  approprlata 
rafarancaa  to  all  tha  organizational  alamanta'  plana.  Tha  davalopmant 
contractor  activity  la  raaponalbla  for  davaloplng  taat  plana  and 
procaduraa  to  affact  an  approprlata  aharing  (aa  daflnad  contractually 
In  tha  SOW)  of  taat  raaulta. 


GLOSSARY! 


RESPONSE  INSTRUCTIONS: 


i:  . 


m 


RESPONSE  RATIONALE: 


5S5E9N5S-5SQ5ii _ 

(COMPLETELY  BISAGREE  >  1.2. 3. 4. 5. 6  ■  COMPLETELY  AGREE) 


Al-31 


I 


QUESTION  DATA  SHEET 


m 

QuMtlon  Nuab«r  SPH(PL>  -  029 

Oy^lSTIQN ;  Tracking  o£  conputar  raaourca  utilization  haa  baan 
adaquataly  planned. 

ACTIVITY(S):  All 

;  Tha  procuraaant  activity  auat  plan  to  hava  tracking  data 
eollaetad.  Thla  plan  ia  put  into  action  through  appropriate  contract 
raquiraaanta .  Tha  davalopaant  contractor  auat  plan  to  iaplaaant  data 
collaetion  procaduraa  which  aatiafy  tha  contractual  raquiraaanta  aa  a 
llQllUS*  Normally  nora  detailed  data  auat  ba  eollaetad  by  tha 
davalopaant  contractor  in  order  to  properly  derive  tha  required 
contract  data  and  adaquataly  aanaga  tha  davalopaant  effort.  Typical 
data  to  ba  collected  ineludaa  aaaory,  central  procaaaing  unit  uaaga, 
and  input/output  throughput.  Tha  aaxiaua  and  ainiaua  valuaa  and 
actual  raaourca  utilization  data  valuaa  era  required.  Thia  data  ia 
naeaaaary  to  dataraina  if  required  margina  of  reaarva  will  ba  mat. 
Sea  AFSCP  B00*4A  for  more  information. 

GLOSSARY: 


RESPONSE  instructions: 


RESPONSE  .rationale  ; 


RESPONSE  SCORE: 


OeXPL^TiLV  ^ISAGr.ES  »  AGrEE: 


QUESTION  DATA  SHEET 


QuMtion  Numb«r  SPn(PL>  ~  030 

9UEgTIQNi  Th«  pro3«ct  softwAr*  bud9«t/cost  v«rl«nc«  (budg«t«d 
«ctu«l}  «pp««rs  to  b*  roooonoblo. 

ACTIVITY (S>;  All 

Oopondlng  upon  tho  poropoctlvo,  all  activitiaa  ara 
raquirad  to  adaquataly  nanaga  tha  aoftwara  budgat/eoat.  At  aach  major 
aanaganant  ravlaw  for  tha  activity*  tha  varianca  batwaan  what  waa 
budgatad  and  what  haa  actually  baan  eonauaad  ahould  ba  analyzed. 
Baaad  upon  tha  known  contractual  changaa  in  raqulraaanta  an 
aaaaaaaant  ahould  ba  aada  whathar  tha  varianca  in  coat  ia  within 
certain  liaita.  Tha  liaita  ahould  probably  ba  aatabliahad  during  tha 
Daaonatration  and  Validation  phaaa  through  a  riak  analyaia.  Saa  AFSCP 
AOO-46  for  aora  information. 

SleSSSARYi 

fV^qf;fr/Coa^  Vqylfnga.  Tha  budgatad  coat  of  aoftwara  taak  work 
<WBS}  parformad**  ainua  tha  actual  coat  of  aoftwara  taak  work 
parf oraad . 

gf gSf gg_yf f 4affga «  Tha  Budgat/Coat  Varianca  divided  by  tha 
actual  coat  of  aoftwara  taak  work  parf  oraad  tiaaa  100. 


RESPOMSE_INSTRUCTIONSi 

F/i:  plua  or  ainua  29k  or  aora  Parcantaga  Varianca 

E/2:  plua  or  ainua  20k  to  29k  Parcantaga  Varianca 

0/3:  plua  or  ainua  19k  or  20k  Parcantaga  Varianca 

C/4:  plua  or  ainua  10k  or  19k  Parcantaga  Varianca 

3/9:  plua  or  ainua  9k  or  10k  Parcantaga  Varianca 

A/6:  plua  or  ainua  Ok  or  9k  Parcantaga  Varianca 

Undarruna  can  ba  aa  big  an  indicator  of  a  problem  aa  an  overrun, 
but  aach  caaa  ahould  ba  carefully  conaidarad  on  ita  own  marita. 

5f2ESl*§S-5AII2!iAtSi 


RESP0MSE_SC0RE2 _ 

“<C0MPLETELY”5Is AGREE  ■  1*2*3*4*9.6  -  COMPLETELY  AGREE) 


QUESTION  DATA  SHEET 


m 

QuM^lon  Nuaibcr  SPH(PL>  -  031 

fiSSSXlQlil  projaeb  aoftwar*  achadulaL/cost  varlanc*  (consumad 
•ehadulad)  appears  bo  ba  raaonabla. 

ACTIVITY<S);  All 

g^V/iyi^X^QW*  Depending  upon  the  parapeetlve,  all  activltiea  are 
required  to  adequately  aenage  the  software  schedule /coat.  At  each 
ae^or  aanagaaent  review  for  the  activity*  the  variance  between  what 
wee  eonauaed  and  whet  hae  actually  been  scheduled  should  ba  analysed. 
Baaed  upon  the  known  contractual  changes  in  requiraaanta  an 
eeeeaeaent  should  be  aade  whether  the  variance  in  coat  is  within 
certain  liaita.  The  liaita  ehould  probably  be  established  during  the 
Deaonatration  and  Validation  phase  through  a  risk  analysis.  See  AFSCP 
SOO-46  for  aore  inforaatlon. 

budgeted  coat  of  software  task  work 
(WBS)  perforaed  ainua  the  budgeted  coat  of  aoftwara  task  work 
scheduled. 

Pf jpgfgj^fgf.VfyjrfgSf «  The  Schedule/Coat  Variance  divided  by  the 
actual  coat  of  software  teak  work  perforaed  tlaea  100. 

RESPQNSE.IMST5UCTI0NSi, 

F/i:  plus  or  ainua  2Sk  or  aore  Percentage  Variance 

jS/2:  plus  or  ainua  20k  to  2Sk  Percentage  Variance 

0/3:  plus  or  ainua  19k  or  20k  Percentage  Variance 

C/4:  plus  or  ainua  10k  or  19k  Percentage  Variance 

B/9:  plus  or  ainua  9k  or  10k  Percentage  Variance 

A/6:  plus  or  ainua  Ok  or  9k  Percentage  Veriance 

Schedule  decreaaee  can  be  ea  big  an  indicator  of  a  problem  as 
increaaea,  but  each  ease  should  be  carefully  considered  on  its 
own  aerita. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

“TcOMPLETELY'DISAGRii  ■  1*2*3, 4*9. 6  ■  COMPLETELY  AGREE) 


Al-34 


QUESTION  DATA  SHEET 

Question  Number  SPM(PL)  ~  032 

9U£SX£S£il  cost  end  schedule  contractusl  reporting  raquirsments 
sppesr  to  bs  sdequste. 

frffTIVITYCS);  All 

dsts  necesssry  to  dstsrnine  the  cost  variance  and 
schedule  variance  inforsation  in  questions  SPH<PL)‘*030  and 
SPK<PL)-031  are  a  minimum  requirement.  AFSCP  800-48  providea  further 
inSormstion.  OoOI  7000.2  is  the  policy  for  financial  aanagement 
reporting  for  the  development  contractor  activity  (appropriate  size 
and  typas  of  contracts) .  AFSCP  173-5  implements  the  policy  of  DoDI 
7000.2  and  provides  specific  criteria  for  the  development  contractor 
activity. 


GLOSSARY: 


5i§EQli§S-Il!5I5liSIISl!§i 


RgSPpNSg  .RATiqwA^g; 


RgSPQNSE  SCOREi _ 

~<c5hPLETELY~0  IS  AGREE  «  1,2, 3, 4, 5, 6  ■  COMPLETELY  AGREE) 


A1-3S 


SOFTWARE  PROJECT  HANAGEMEMT  ORGANIZATXON  STRUCTURE 


Th«  quMtiona  SPH<0S}-001  through  SPH(OS)<-O19  addroaa  adaquacy 
o£  aoftwara  projact  aanagaaant  organization  atructura  for  tha 
procuranant*  davalopnant  contractor,  and  operation  aupport 
aetlvitiaa.  Projact  aanagaaant  organization  atructxira  la  aatabliahad 
ao  that  tha  functional  raquiraaanta  of  a  project  can  ba  more 
e£f actively  aeeoapliahed.  Thia  organization  atrueture  includaa  tha 
phyaical  ralationahip  aaong  the  organization  eleaenta,  tha  logical 
atrueture  which  ralatea  tha  organization  alaaanta  to  tha  projact'a 
functional  raquiraaanta,  and  the  atability  and  capability  of  the 
peraonnel  which  ataff  tha  organization. 

Charactariatica  which  affect  tha  organization  atructura  are 
number  of  intarnal  intarfacea,  aiza  of  organization,  atability  of  tha 
phyaical  atructura,  continuity  and  capability  of  projact  paraonnal, 
and  capability  of  tha  phyaical  organization  to  affectivaly  handle 
reaponaibilitiaa  inharant  in  the  aoftwara  project  work  breakdown 
atrueture  taaka. 


QUESTION  DATA  SHEET 


Question  Number  SPM(OS>  >  001 

pUgST^QN :  Th«  software  raquirananta  hava  baan  adaquataly  allocatad  to 
alaaanta  of  a~Work  firaakdown  Structura  (WBS) . 


ACTIVITY <S) :  All 

;  Tha  WBS  ahould  elaarly  indieata  thoaa  taak  araaa  whara 
aoftwara  ralatad  raqulraaanta  ara  addrasaad.  A  bracaabllity  matrix 
ahould  elaarly  Indieata  how  thoaa  raqulraaanta  hava  baan  nappad  to 
tha  WBS  alaaanta.  All  aetivitlaa  (proeuranant,  davalopaant 
eontraetor*  oparation  aupport)  ara  Involvad  in  aona  aapact  of  tha 
WBS#  aoftwara  raqulraaanta#  and  tha  alloeatlon  proeaaa. 


glossary: 

Contraetual  ayataa  raqulraaanta  which  hava 
baan  allocatad  to  aoftwara  funetlona.  Uaually  found  in  .  a 
ayataa/aagaant  apaelf leatlon  and/or  a  funetlonal  davalopaant 
apaelf leatlen • 

-  ,  8rf  f  -  sscyssy  Produet-orlantad  hlararehical 
daflnltlon  of  all  taaka  to  ba  parforaad  and  aeeountad  for  in  tha 
eouraa  of  tha  projaet  davalopaant. 


5§§S2l!2S-ll!§I5USI121!5l 


F/i:  No  WBS  axiata#  no  aoftwara  raqulraaanta  apaelf ication 

axlata.  and/or  thara  la  no  avidanca  of  an  allocation  of 
aoftwara  raqulraaanta  to  WBS  alaaanta. 


RESPONSE  RATIONALE: 


RESPONSE^SCOgli  _ 

o5iipLiT!iL7“5ioAGSEH  » 


,^.G.4.3.o  -  ■:CMPLE“EL7  AGPS-Z: 


QUESTION  DATA  SHEET 


OuMtion  Nuabcr  SPHCOS)  -  002 


flUESXIQSil  •oftw«r«  task«  «r«  clsarly  ld«ntifi«d  in  th« 

WBS. 


AgiIVITY<S>  j  All 


J  JpN ;  Typical  aoftwar*  ta«k«  include  pro^ac-t  managanant, 
davalopaant#  docuaantatlon,  taab*  anvironaant  avolution, 

conSiguration  aanaganant#  quality  aaauranca*  accaptanca,  and 
tranafar.  All  aoftwara  ralatad  taaka  should  ba  aaparataly  idantifiad 
from  hardwara  ralatad  taaka  aa  much  aa  ia  poaaibla.  Tha  mora 
aceurataly  aueh  taaka  can  ba  trackad  and  raportad*  tha  nora  likaly 
aarly  problaa  araaa  can  ba  idantifiad  and  raaolvad. 


CLgSS^RYi 

^O^trWqra  Task.  Projact  task  whoaa  primary  function  ia  ralatad  to 
tha  production  and/or  dalivary  of  a  aoftwara  product. 


RESPONSE  INSTRUCTIONS: 


RESPONSE  RATIONALE: 


RESPONSE_SCgREi 

’’TcOMPLEfiLY'oiSAGRii  «  1.2. 3. 4. 5, 6  ■  COMPLETELY  AGREE 3 


Ai-sa 


QUESTION  DATA  SHEET 


QuMtion  Number  SPKCOS)  -  003 

(m^gSTION:  Th«  .k«y  project  pereonnel  end  their  aeelgnmente  in  relation 
to  the^wis  software  related  taeke  ere  clearly  identified. 


ACTIVITYCS) ;  All 

order  to  properly  identify  reeponaibilitiea  end 
coeeunicetlon  channels  it  is  necessery  to  have  the  key  project 
personnel  for  all  activities  identified  along  with  their  areaa  of 
responsibility  and  their  reletionship  to  the  WBS. 


GLOSSARY ; 

K§x^££S2SSi^SSI§SSDSlj.  Project  sanagera*  task  managers,  task 
technical  leaders  for  all  activities. 


RESPONSE  INSTRUCTIONS; 


RESPONSE  RATIONALE! 


■.E2?0NSS_3CCHEj, 

”7cCMPtiTSLY“5lSAGRis  ~  1.2. 3. 4. 3. 3  « 


CC.'^PLSTELY  agree: 


Al-39 


QUESTION  DATA  SHEET 


QuMtion  Nuab^r  SPH(OS>  -  004 

9UE?TI0N;  Tha  coordination  of  aodif icationa  to  tha  WBS  among  all 
aetivitlaa  haa  baan  adaquata. 


All 

\  Whanavar  tha  WBS  ia  nodiflad  thara  ia  potantlal  to  hava 
a  aignif leant  affaet  upon  all  aetivitlaa.  Thla  affact  nay  ba  in  tha 
fora  of  a  modification  In  aehadula*  laval  of  affort,  dallvarabla 
product*  functional  capability*  and/or  ayatam  Intarfaea.  A  machanlan 
to  coordinata  aueh  ehangaa  ia  naeaaaary  in  ordar  to  aaka  aura  all 
potantlally  affactad  partiaa  ara  proparly  eonaultad.  Without  auch  a 
maehaniaa*  ehangaa  can  ba  mada  without  auch  conaultation  or  parhapa 
without  baing  proparly  raflactad  in  tha  WBS. 


glossary; 


R^?P0WSE  INSTRUCTIONg; 


RESPONSE  RATIONALE! 


RESPONSE  SCORE:  _ 

“TcOMPLETELY'BiS AGREE  •  1*2,3*4,S*6  ■  COMPLETELY  AGREE) 


QUESTION  DATA  SHEET 


QuMtion  NuiRb«r  SPH(OS)  -  005 

OUESTIQN^  Th«  procur«»ttnt  p«raonn«l  staffing  has  had  continuity 
throughout  ths  software  life  cycls  phassa. 


^9TIVITY<S) :  Proeursaant 

staffing  continuity  is  dstsrsinsd  by  ths  rats  of 
tumovsr  of  psrsonnsl  during  and  across  ths  lifs  cycls  phasss.  If  ths 
aass  psrsonnsl  (or  at  Isast  a  rsssonabls  ratio  of  ths  sans  psrsonnsl) 
ars  not  svailabls  from  phass  to  phsss«  than  thsrs  is  liksly  to  bs  a 
perturbation  in  ths  schaduls,  cost*  functional  rsquirsnsnts,  and 
quality  of  ths  dslivsrabls  products.  Turnover  of  key  psrsonnsl  should 
bs  nininal  with  no  sharp  variations.  See  AFSCP  S00-4d  for  sore 
information. 


glossary; 

Lack  of  turnover  and  sharp  change  in  psrsonnsl  staff . 


5SS£S!f§i_iNSTRUCTigMSi 

F/i:  son  or  mors  turnover  during  or  bstwsan  any  one  phass: 

(Concept*  Demonstration*  Development*  Deployment} 

S/2:  son  to  son  turnover  during  or  between  any  one  phass 

0/3:  30n  to  40n  turnover  during  or  between  any  one  phass 

C/4:  20n  to  30n  turnover  during  or  between  any  one  phase 

B/S:  ion  to  20n  turnover  during  or  between  any  one  phase 

A/6:  on  to  ion  turnover  during  or  between  all  phases 

RESPONSE  RATION/;*.^; 


?!£3P0iiS£_iC0ftS: 

’TcOKPlItSL  Y“3IoAGi2£">”*. . 2 . 3 , 4 . 5 ,  S  « 


33  M  P  L  2T2!.  ?  .*  G  ?  2” : 


Al-41 


QUESTION  DATA  SHEET 


QuMtion  Nu»b«r  SPHCOS)  -  006 

9US2XIQ£l  ratio  of  oxporioncod  procuronont  projoct  poraonnol  to 
tho  total  nuabar  of  projact  paraonnal  haa  baan  adaquata  throughout 
tha  aoftwara  Ufa  cycla  phaaaa. 

ACTIVITY <S> :  Proeuraaant 

gjCg^y^XyON ;  Ona  kay  to  davaloplng  and  aupporting  aoftwara  la  to  hava 
axpariancad  paraonnal,  aapaelally  In  tha  kay  laadarahlp  poaltiona. 
Exparlanea  with  tha  aubjact  ayataa,  aiailar  ayataaa,  tachnologlcally 
aiailar  problaaa,  tha  aanagaaant  problaaa  of  aiailar  ayatana,  and  tha 
Intarfaeaa  with  tha  aubjact  ayataa  raaulta  in  battar  aanagad, 
hlghar  quality  aoftwara  producta.  Exparlanead  paraonnal  alao  ahortan 
tha  laarnlng  eurva  for  laaa  axpariancad  paraonnal .  .  Saa  AFSCP  800-48 
for  aora  Inforaatlon. 

guqssARY; 

Paraonnal  who  hava  an  axtanalva  hlatorlcal 
parapactlva  of  tha  aiibjaet  ayataa  and  Ita  aoftwara  raqulraaanta,  aa 
wall  aa  tha  tachnleal  axpartlaa  and/or  aanagaaant  axpartlaa  raqulrad 
to  afflelantly  laplaaant  tha  raqulrad  aoftwara  aolutlona. 


RESPONSE  INSTRUCTIONS; 

F/i:  10%  or  laaa  axpariancad  paraonnal  during  any  ona  phaaa 
(Concapt,  Daaonatratlon,  Oavalopaant,  Daployaant) 

E/2:  10%  to  20%  axpariancad  paraonnal  during  any  ona  phaaa 

0/3:  20%  to  30%  axpariancad  paraonnal  during  any  ona  phaaa 

C/4:  30%  to  40%  axpariancad  paraonnal  during  any  ona  phaaa 

B/S:  40%  to  S0%  axpariancad  paraonnal  during  any  ona  phaaa 

A/6:  S0%  or  aora  axpariancad  paraonnal  during  all  phaaaa 

RESPONSE  RATiqNAVf: 


RESPONSE^SCORgi 

“<c5MPLETELY“5lSAGRii’«"l,2,3,4,5,6  ■  COMPLETELY  AGREE) 


QUESTION  DATA  SHEET 


QuMtlon  Nuiib«r  SPHCOS)  -  007 

SlIESIIQNl  Th«  nuaib«r  of  proeuromont  poraonnol  has  boon  odoquato 
throughout  tho  ooftworo  llfo  cyelo  phoooo. 


ACTIVITYiSll  Procurooont 

nuobor  of  poroonnol  should  bo  sufflclont  to  support 
tho  rosponsibllitiss  roqulrod  by  tho  program.  Sufflclont  nunbor  la 
dopondont  upon  matching  tho  oxporlsnco*  worklood*  and  productivity 
roqulroaonts .  Typically#  fouor  parsons  aro  roqulrod  during  tho 
eoncopt  phaao#  mors  porsons  during  tha  daaonstratlon  and  than 
davalopaant  phasaa,  and  gradually  fawar  parsons  prior  to  tha 
doploymant  phasa  of  tha  projact.  Ona  mathod  of  datarmlnlng 
"sufflclont**  Is  to  assumo  tho  allocatod  numbar  of  porsonnal  la  tho 
most  optimum,  and  dotoralno  tha  ratio  of  actual  to  allocatad 
parsonnal.  Culdallnas  for  an  evaluation  rasponso  basod  on  this  ratio 
aro  glvon  balow.  Saa  AFSCP  A00**46  for  mora  Information. 

StSSSASii 

Btfgfegg, , EgggUgtgfgir, ?trf ^  1  count  of  porsonnal  directly 
rasponslbla  for  procuraaont  functions  ralativa  to  tho  subject  systam. 
This  Includes  direct  software  project  managaaiant,  tochnlcsl  staff, 
and  support  staff  .  Only  those  staff  positions  dlroctly  allocatod,  or 
through  dlroct  asslgnaont  of  an  allocatod  position,  should  bo 
consldorod.  If  nacassary,  a  "full  time  aqulvalont"  (o.g.,  part  of  a 
position  which  Is  shared  among  ona  or  mora  other  ayatoms>  voluo  can 
bo  used. 


F/i:  ox  to  sox  of  allocotod  lood  during  any  llfo  cycle  phasa 
(Concept,  Oomonatratlon,  Development,  Doploymant) 

S/2:  SOX  to  60X  of  allocatad  load  during  any  life  cycle  phasa 

0/3:  60X  to  70X  of  allocatad  load  during  any  llfo  cycle  phasa 

C/4:  70X  to  sox  of  allocatad  load  during  any  Ufa  cycle  phasa 

B/S:  SOX  to  90X  of  allocatad  lood  during  ony  llfo  cycle  phasa 

A/S:  90X  or  xora  of  allocatad  load  during  all  Ilfs  cycle  phases 

SiSPCfNSs-SgllSNAtSi 


RESPONSE  SCORE:  _ 

"(C0MPLETiLY~'5is AGREE  ■  1,2,3,4,S,6  -  COMPLETELY  AGREE) 


QUESTION  DATA  SHEET 


QuMtion  Nuabar  SPH(OS)  -  OOd 

fiUESIlQlil  Tha  davalopaant  contractor  parsonnal  staffing  has  had 
continuity  throughout  tha  software  life  cycle  pheeea. 


Development  Contractor 

staffing  continuity  la  deterelned  by  the  rate  of 
turnover  of  personnel  during  and  across  the  life  cycle  phases.  If  the 
same  personnel  (or  at  least  a  raaaonable  ratio  of  tha  same  personnel) 
ere  not  available  from  phase  to  phase,  then  there  is  likely  to  be  a 
perturbation  in  the  schedule,  coat,  functional  raquirementa,  and 
quality  of  the  deliverable  products.  Turnover  of  key  personnel  should 
be  minimal  with  no  sharp  variations.  See  AFSCP  A00’-4d  for  more 
information . 

glossary: 

S9Q^4nuitv.  Lack  of  turnover  and  sharp  change  in  personnel  staff . 

(SXk 


5I2£2S2I-IJ!§ISUSI5Si!§i 

F/i:  90m  or  mors  turnover  during  or  between  any  one  phase: 

(Concept,  Demonstration,  Development,  Deployment) 

E/2:  40K  to  90k  turnover  during  or  between  any  one  phase 

0/3:  30k  to  40k  turnover  during  or  between  any  one  phase 

C/4:  20k  to  30k  turnover  during  or  between  any  one  phase 

1/9:  10k  to  20k  turnover  during  or  between  any  one  phase 

A/6:  '  Ok  to  10k  turnover  during  or  between  all  phases 

RESPONSE  RATIONALE: 


RESPONSE  SCORE:  _ 

“(CSRPLETELY-DISAGRii  ■  1,2, 3, 4. 9, 6  ■  COMPLETELY  AGREE) 


QIJESTION  DATA  SHEET 


QuMtlon  NuMbcr  SPH(OS>  -  009 

fiSSSIISNl  Th«  ratio  of  axparlancad  davalopaant  contractor  pro 3 act 
parsonntl  to  tha  total  nuabar  of  pro^act  paraonnal  haa  baan  adaquata 
throughout  tha  aoftwara  Ufa  cycla  phaaaa. 

dSTI2^1I2i92l  Davalopaant  Contractor 

davaloping  and  aupportlng  aoftwara  la  to  hava 
axparlancad  paraonnal,  aapaclally  In  tha  kay  laadarahip  poaltlona. 
Exparlanca  with  tha  aubjact  ayataa,  alallar  ayataaa,  technologically 
alallar  problaaa,  tha  aanagaaant  problaaa  of  alallar  ayataaa,  and  tha 
Intarfacaa  with  tha  aubjact  ayataa  raaulta  In  battar  aanagad,  higher 
quality  aoftwara  producta.  Exparlancaid  paraonnal  alao  ahortan  tha 
learning  curve  for  laaa  axparlancad  paraonnal.  Saa  AFSCP  800-4d  for 
aora  Inforaatlon. 


GLOSSARY: 

gypf r4f ngf j  Pf f f ^ j  Paraonnal  who  hava  an  axtanalva  hiatorlcal 
parapactlva  of  tha  aubjaet  ayataa  and  Ita  aoftwara  raqulraaanta,  aa 
wall  aa  tha  technical  axpartlaa  and/or  aanageaant  axpartiaa  raqui-rad 
to  efficiently  laplaaant  tha  required  aoftwara  aolutlona. 


5£§EQll§i-IN§TRUCTI0NSi 

F/i:  lOx  or  laaa  axparlancad  paraonnal  during  any  ona  phaaa 
< Concept,  Deaonatratlon,  Davalopaant,  Daployaant) 

E/2:  lOX  to  20X  axparlancad  paraonnal  during  any  ona  phaaa 

D/3:  20X  to  30X  axparlancad  paraonnal  during  any  ona  phaaa 

C/4:  30X  to  40X  axparlancad  paraonnal  during  any  ona  phaaa 

B/S:  40X  to  90X  axparlancad  paraonnal  during  any  ona  phaaa 

A/6:  50X  or  more  axparlancad  paraonnal  during  all  phaaaa 

RESPONSE  rationale: 


RESPONSE_SCgREi  _ 

~7c0MPLiTELY~DiSAGREE  >  1,2, 3, 4, 9, 6  ■  COMPLETELY  AGREE) 


Al-45 


QUESTION  DATA  SHEET 


QuMtlon  Nu»b«r  SPHCOS)  -  010 

OUESTIQgl  Th«  nuKb«r  of  dovolopnont  contractor  parsonnal  haa  baan 
adaquata  throughout  tha  aoftwara  Ufa  cycla  phaaaa. 


ACTIVITY < 8) :  Oavalopaant  Contractor 

nuabar  of  paraonnal  ahould  ba  aufficlant  to  support 
tha  raaponalbllltiaa  raquirad  by  tha  program.  Sufflclant  nuabar  la 
dapandant  upon  matching  tha  axparlanca.  workload*  and  productivity 
raquiraaanta .  Typically*  fawar  parsons  sra  raquirad  during  tha  aar ly 
raquiraaants  phasa*  aora  parsons  during  tha  dasign  and  iaplaaantation 
phasas*  and  gradually  fawar  parsons  prior  to  tha  production  phasa  of 
tha  projact.  Thara  sra  faw  good  guidalinas  as  to  what  is  “aufficiant” 
for  s  davalopaant  contractor.  Tha  usa  of  cost  astiastion  aquations  as 
proposad  In  tha  lltaratura  (a.g.*  B.  Boaha's  COCOHO  modal)  Is 
possibla*  but  not  vary  siapla  to  apply.  Ona  guldalina  la  to  maka  sura 
sharp  incraasaa  or  dacraasas  In  tha  nuabar  of  parsonnal  do  not  occur. 
Saa  AF8CP  BOO-AB  for  aora  Information. 

Tha  count  oi 

parsonnal  diractly  rasponsibla  for  davalopaant  contractor  functio#w 
ralativa  to  tha  subjact  systaa.  This  ineludas  diract  aoftwara  proja^ 
aanagaaant*  tachnieal  staff*  and  support  staff.  Only  thoaa  staff 
positions  diractly  assignad  should  ba  considarad.  If  nacasaary*  a 
"full  tiaa  aquivalant"  (a.g.*  part  of  a  position  which  la  sharad 
among  ona  or  aora  othar  aystaaa)  valua  can  ba  usad. 


RESPONSE  INSTRUCTIONS: 


SiS£91jSS-5AXiSNAt£2 


giSPONSE  SCORE:  _ 

'*(C0HPLiTiLY~5ISAGREi  >  1*2*3*4*5*6  -  COMPLETELY  AGREE) 


QUESTION  DATA  SHEET 


QuMtlon  Nu»b«r  SPH(OS)  -  Oil 

OySSTION:  Th«  oparatlon  mupport  p^rconnal  staffing  has  had  continuity 
throughout  tha  software  lifa  cycla  phaaaa. 


ACTIVITY <S> ;  Oparation  Support 

staffing  continuity  is  datarainad  by  tha  rata  of 
turnovar  of  paraonnal  during  and  across  tha  lifa  cycla  phaaaa.  If  tha 
saaa  paraonnal  <or  at  laaat  a  raaaonabla  ratio  of  tha  sans  paraonnal) 
ara  not  availabla  from  phaaa  to  phaaa»  than  thara  ia  likaly  to  ba  a 
parturbation  in  tha  schadula*  coat,  functional  raquiraaanta,  and 
quality  of  tha  dalivarabla  products.  Turnovar  of  kay  paraonnal  should 
ba  ainiaal  with  no  sharp  variations.  Saa  AFSCP  dOO-46  for  nora 
information. 


GLOSSARY: 

SSQiiONilY.1  l-*ck  of  turnovar  and  sharp  changa  in  paraonnal  staff . 


RESPgjfSE^INSTRUCTIONSi 

F/i:  SOk  or  aora  turnovar  during  or  batwaan  any  ona  phaaa: 

(Concapt,  Oamonatration.  Oavalopaant,  Daploymant) 

S/2:  40k  to  90k  turnovar  during  or  batwaan  any  ona  phaaa 

0/3:  30k  to  40k  turnovar  during  or  batwaan  any  ona  phaaa 

C/4:  20k  to  30k  turnovar  during  or  batwaan  any  ona  phaaa  * 

•/9:  10k  to  20k  turnovar  during  or  batwaan  any  ona  phaaa 

A/6:  Ok  to  10k  turnovar  during  or  batwaan  all  phaaaa 

RESPONSE  .RATION ale : 


RESPONSE  SCORE:  _ 

~7cOHPLiTiLY'DiSAGREE  «  1,2, 3, 4. 9. 6  >  COMPLETELY  AGREE) 


QUESTION  DATA  SHEET 


QuMtion  Mtmbttr  SPM(08>  -  012 


QUESTION!  Th«  ratio  of  oxporioncod  oporatlon  support  product 
psrsonnsi  to  ths  totsl  numbsr  of  projsct  psrsonnsl  hss  boon  sdsqusts 
throughout  ths  softwsrs  Ilfs  eycls  phasss. 


ACTIVITY <S) :  Opsratlon  Support 


gj(g^jlAT|ON!  0ns  ksy  to  supporting  softwars  Is  to  hsvs  axparlsncad 
psrsonnsl#  sspsclally  In  ths  ksy  Isadsrshlp  positions.  Expar lanes 
with  ths  subjact  systss#  similar  aystama#  tsehnologleally  similar 
problams#  tha  mansgamant  problams  of  similar  systams#  and  tha 
Intarfaeas  with  tha  subjact  systam  rasults  In  bsttsr  managad#  hlghar 
quality  softwars  ravlslons.  Exparlanead  support  parsonnal  also 
shortan  tha  laarnlng  eurva  for  lass  axparlsncad  support  parsonnal. 
Saa  AFSCP  800-<iA  for  mors  Information.  ■ 


GLOSSARY! 

typf pSf Etgl9gDtA.t  Parsonnal  who  havs  an  axtsnsiva  historical 
psrspaetlva  of  tha  subjaet  systam  and  Its  aoftwsra  raquiramants *  as 
wall  as  tha  taehnlesl  axpartlss  and/or  mansgamant  sxpsrtlaa  raqulrad 
to  Sfflclantly  Implsmsnt  modifications  to  ths  dsllvarad  software 
products.  4E 


RESPONSE  INSTRUCTIONS ! 

F/l!  lOX  or  lass  axparlsncad 
< Concept »  Oamonstratlon# 
E/2!  lOX  to  20X  axparlsncad 

0/3!  20X  to  30X  axparlsncad 

C/4:  30X  to  40X  axparlsncad 

3/S:  40X  to  SOX  axparlsncad 

A/6!  SOX  or  aora  axparlsncad 


parsonnal  during  any  one  phase 
Davalopmant#  Daploymant) 
parsonnal  during  any  one  phase 
parsonnal  during  any  one  phase 
parsonnal  during  any  one  phase 
parsonnal  during  any  ona  phase 
parsonnal  during  all  phases 


RESPONSE  RATIONALE: 


BSSESSSs-SSQSEi  _ 

”'iTTL7"'?T:;AGRSi 


t  •  w  •  O  ^ 
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*  •  1.  •  • 

iMiwL. 


QUESTION  DATA  SHEET 


QuMtion  Number  SPH(OS}  -  013 

QU£SXI9lil  number  of  operation  eupport  personnel  has  been  adequate 
throughout  the  software  life  cycle  phases. 


AC;XiyiTY<S),  t,  Operation  Support 

number  of  personnel  should  be  sufficient  to  support 
the  responsibilities  required  by  the  program.  Sufficient  number  .  is 
dependent  upon  matching  the  experience*  workload*  and  productivity 
requirements.  Typically*  fewer  operation  support  persons  are  required 
during  the  concept  phase*  more  persona  during  the  demonstration  and 
then  development  phases*  and  maximum  personnel  during  the  deployment 
phase  of  the  project.  One  method  of  determining  “sufficient'*  is  to 
assume  the  allocated  number  of  personnel  is  the  most  optimum*  snd 
determine  the  ratio  of  actual  to  allocated  personnel.  Guidelines  for 
an  evaluation  response  based  on  this  ratio  are  given  below.  See  AFSC? 
3C0*4d  for  more  information. 

GLOSSARY^ 

Ntf mbf A 9S - Stf Egggt- Pf gf 90Sf  ^ «  The  count  of  personnel 
directly  responsible  for  operation  support  functions  relative  to  the 
subject  system.  This  includes  direct  software  project  management* 
technical  staff*  and  support  staff.  Only  those,  staff  positions 
directly  allocated*  or  through  .  direct  assignment  of  an  allocated 
;.';«itior. ,  .ihauld  be  considered.  If  necessary*  a  "f-all  ti;!;a 
.‘.c-.ivalent"  (e.g.,  part  of  a  position  which  is  shared  among  one  or 
othe.r  systems)  value  can  be  used. 


t  tS  ^  •  .  >  .«  W  •  «  W  «N  M  « 


T/  •*  • 
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load 

during 

any 
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cycle 

phase 

( Concept ,  Oe.monatration, 

Oevelop.ment » 
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50J; 

to 
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of 
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load 
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life 

cycle 
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0/3: 

60X 

to 

70X 

of 

allocated 

load 

during 

any 

life 

cycle 

phase 

C/4: 

70X 

to 

30X 

of 

allocated 

load 

during 

any 

life 

cycle 

phase 

B/S: 

80m 

to 

90X 

of 

allocated 

load 

during 

any 

life 

cycle 

phase 

A/6: 

90X 

or 

more 

of 

allocated 

load 

during 

all 

life 

cycle 

phases 

SSSPCMSS  PATICMALSJ 


“7cCh?L2?iLy"5lSAGiii”«”l,2,3,4*5,6  ■  COKPLETELY  AGREE) 
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QUESTION  DATA  SHEET 


Question  Number  SPH(OS} 


ai-ESTrON:  The  internal  interfaces  among  procurement  organization 
elementa  have  been  adequate. 


ACTIVITY (S):  Procurement 

Internal  organization  elementa  might  include  the  program 
office, 'configuration  management  organization#  development  teat  and 
evaluation  agency#  operational  teat  and  evaluation  agency# 
independent  verification  and  validation  organization#  and  various 
intoroervice  elementa  am  appropriate.  Characteriatics  of  the 
interfacjs  to  be  aaaesoed  include  proper  decision  process  information 
fl<?w#  ef fectlvenees  of  information  flow#  i<;\nd  adherence  to  regulations 
and  guLdelineu  for  interface  responsibility. 

ftr  • 


RSSPC?!Sg_gATIONA^g! 


•••••  ••  W  ww  •  ••  • 

“”c5x?LiTiLY"5:sAGRii” »”i , 2  #  3  #  4 


.5, 


6  >  COMPLETELY  AGREE) 


QUESTION  DATA  SHEET 


QuMtlon  Number  SPH(OS>  -  OIS 

QUESTION;  The  Internal  Interfecee  among  development  contractor 
organization  elementa  have  been  adequate. 


ACTIVITY {3> ;  Development  Contrector 

Internal  organization  elementa  might  include  the  project 
management f^configurat ion  management  group,  project  technical  ataff, 
hardware  and  aoftware  groupa,  quality  aaaurance  group,  independent 
teat  group,  contract  management,  and  corporate  management. 
Characteriatica  of  the  interfacea  to  be  aaaeaaed  include  proper 
deciaion  proceaa  information  flow,  effectiveneaa  of  information  flow, 
and  adherence  to  regulation^,  and  guidelinea  for  interface 
raaponaibi 1 ity . 


•?ss?CNss  instructions: 


5S2S2Ji2S-5AIISl!AkSi 


RESPONSE  SCORE: 

“c5?.?LSTiLY“5:SAGiii”  1,2,3,4,S,6  ■  COMPLETELY  AGREE) 


QUESTION  DATA  SHEET 


QuMtlon  NuBb«r  SPM(OS>  >  016 


QUESTION:  Th«  Intarnal  lnt«r£ac«s  among  opmrmtlon  support 

organization  alamanta  hava  baan  adaquata. 


ACTIVITYi^L I  Oparatlon  Support 

Internal  organization  alamanta  might  include  the  project 
managamant*  configuration  management  group,  project  technical  staff, 

group,  independent 
and  using  command 


hardware  and  software  groups,  quslity  assurance 
test  group,  contract  management,  and  supporting 
management.  Characteriatlca  of  the  interfaces  to  be  assessed  include 
proper  decision  process  information  flow, 
information  flow,  and  adherence  to  regulations 
interface  responsibility. 


effectivaneaa  of 
and  guidelines  for 


G’ 0«SA!?Y; 


uI§S2SI»»_IS5I522II2i!2i 


RESPONSE  RATIONALE: 


RESPONSE  SCORE: 

”7cONPLi7iLY“5lSAGRii' 


>  1,2, 3, 4, 3,6  >  COHPLETELY  AGREE) 


QUESTION  DATA  SHEET 

QuMtlon  Number  SPH(OS)  -  017 

OTJSSTION ;  The  procurement  phyelcel  orgenlzetion  etructure  hes  been 
adequate . 


Procurement 

project  phyeicel  structure  is  typically  represented 
in  sn‘'orgsnization  chert.  The  phyeicel  orgsnizstion  should  have  s 
logical  relationship  to  the  project  WBS>  with  the  specific  physics! 
organization  elements  having  a  well-defined  functional  responsibility 
for  a  part  of  the  UBS.  The  software  parts  of  the  physical  structure 
■should  be  clearly  identified  and  adequate  to  accomplish  the 
r«spcnaibilities  inherent  in  the  WBS. 


vS3?QNS5 ..  IW3XS2CT;QN§2 

r/i:  Nonphysical  organization  chart  or  equivalent  is  available^ 
or  it  is  not  current. 


1S3P0N3E  -NATIONALS: 


7;"3?CM3S 


_-3cgsii _ 

TiLY“5tSAGiis  ■  1,2,3,4,S,6  «  COMPLETELY  AGREE) 


Al-53 


QUESTION  DATA  SHEET 


QuMtlon  Numbar  SPIf(OS)  -  Old 

QUESTION;  Th«  d«v*lopm«nt  contractor  physical  organization  atructura 

has  baan  adaquata. 

Oavalopmant  Contractor 

pro^act  physical  atructura  ia  typically  rapraaantad 
in  an  organization  chart.  Tha-  physical  organization  should  hava  a 
logical  ralationahip  to  tha  projact  WBS>  with  tha  apacific  physical 
organization  alaaanta  having  a  wall-dafinad  functional  raaponaibility 
for  a  part  of  tha  WBS.  Tha  software  parts  of  tha  physical  atructura 
should  ba  claarly  idantifiad  and  adaquata  to  accomplish  tha 
raaponaibilitiaa  inharant  in  tha  WBS. 


y^eggAgYj 


T/li  No  physical  organization  chart  or  aqulvalant  ia  availabla. 
or  it  ia  not  currant. 


R^SPONg^  RATIONALE; 


S=§?g>is|_3cgRE2 _ 

"‘cOMPLSTiLY’DISAGREE’-  1.2, 3, 4. 9, 6  -  COMPLETELY  AGREE) 


QUESTION  DATA  SHEET 

QuMtion  Number  SPM(OS>  ~  019 

oySSTIQN ;  The  operetlon  support  phyelcel  orgsnlzatlon  structure  has 
been  adequate. 


*  Operation  Support 

CypLANATION*  The  project  phyaieel  atruetxire  la  typically  represented 
in  an  organization  chert.  The  physical  organization  should  have  a 
logical  relationship  to  the  project  WBS»  with  the  specific  physical 
organization  eleaenta  having  a  wall-defined  functional  reaponaibility 
for  a  part  of  the  WBS.  The  software  par  .a  of  the  phyaical  structure 
should  be  clearly  identified  and  adequate  to  accoapliah  the 
responalbilitiea  inherent  in  the  UBS.^ 


(glOS^AHY  t 


5I5221!5I-Il!5IS2SII92!5i 

~r7lT*No  physical  organization  chart  or  equivalent  is  available, 
or  it  la  not  current. 


RESPONSE  RATIONALE: 


RESPONSE  SCORE:  _ 

“cCOMPLSTiLY”DISAGREE  ■  1.2, 3. 4, 9. 6  ■  COMPLETELY  AGREE) 


Al-99 


SOFTWARE  PROJECT  MANAGEMENT  DESIGN  METBOOS 


Th«  quMtlon*  SPH<OM)-O01  through  SPM(DM>-Oia  addrosa  adaquacy 
of  aoftwara  projact  aanagaaant  daalgn  aathoda  for  tba  procuraaant, 
davalopaant  contractor*  and  operation  aupport  activitiaa.  Project 
managaaant  daalgn  aathoda  are  aatabllahad  ao  that  the  funct.ional 
raquiraaanta  of  a  project  can  be  more  afflciantly  tranacribad  into  an 
laplaaantad  product.  The  daaign  aathoda  iaeluda  atandarda* 
convantiona*  ragulatlona*  direct! vaa*  aoftwara  language*  review 
aathoda*  high  level  rapraaantation  ‘  tachniquaa  and  aathodologiaa* 
autoaatad  daalgn  alda*  and  ao  forth. 

Thera  are  generally  two  aapacta  of  a  daaign  aathod  which  need  to 
ba  evaluated.  Firat*  doaa  the  daaign  aathod  facilitate  production  of 
high  quality  aoftwara  within  Halted  available  raaourcaa?  Second*  can 
the  daaign  aathod  ba  tranaitionad  to  the  aoftwara  aupport  activity 
for  the  evolution  of  the  aoftwara  producta  during  poet  daployaant 
aupport? 

The  proeuranant  activity  ia  raaponaibla  for  aaauring  that 
adequate  daaign  aathoda  have  bean  required  through  the  PHP*  CDRL* 
SOW*  and  any  other  proeuranant  docuaanta.  Tha  proeuranant  activity  ia 
alao  raaponaibla  for  undaratanding  tha  nature  and  iaportanea  of  tha 
davalopaant  contractor  daaign  aathoda  and  tha  affaeta  which  aoftwar^ 
daaign  tradaoffa  night  have  upon  tha  affective  iaplaaantation  of 
daairad  ayataa.  Tha  proeuranant  activity  ahould  have  ita  own  aathoda 
of  aaaaaaing  whether  ayataa  raquiraaanta  and  daaign  apacif ieationa  of 
thoaa  raquirananta  are  conaiatant*  and  whether  tha  daaign  adequately 
iaplamanta  tha  raquiraaanta. 

Tha  davalopaant  contractor  activity  ia  raaponaibla  for 
aatabliahing  daaign  atandarda  appropriate  for  tha  aoftwara  being 
davalopad  ao  that  an  operationally  affective  and  aupportabla  ayatam 
is  produced.  Daaign  rapraaantation  tachniquaa  and  autoaatad  daaign 
aida  ahould  ba  appropriate  for  davalopaant  of  tha  aoftwara  daaign  in 
an  efficient  manner  and  for  tranaition  to  tha  operation  aupport 
activity. 

Tha  operation  aupport  activity  ia  raaponaibla  for  making,  aura 
tha  davalopaant  contractor  ia  required  to  uaa  daaign  aathoda  which 
can  ba  affectively  uaad  during  poat  daployaant  aupport.  Such  daaign 
aathoda  may  require  training  for  tha  aupport  activity  paraonnal.  Tha 
operation  aupport  activity  alao  haa  tha  raapbnaibility  to  aaaura  that 
tha  aupport  anvirenaant  la  to  ba  auppliad  with  all  tha  nacaaaary 
daaign  aida  to  affectively  accoapliah  aoftwara  aupport. 


QUESTION  DATA  SHEET 


QuMtion  Number  SPHCDH)  -  001 

QUESTION;  The  procurement  deelgn  enelyeia  atudiea  have  provided 
edequete'dealgn  guldellnee  for  the  development  contractor. 


ACTIVITycg),  f.  Procurement 

Pff WAff *  DMiQn  enelyeia  atudiea  include  feaaibility  atudiea  on 
the  tiae  o£  computer  reaourcea»  tradeoff  atudiea  concerning 
programming  language  and  inatruction  aet  architacture  aelectiona* 
alternate  approachea  for  implementing  aecurlty  requiramenta , 
altarnata  approachea  for  achieving  operationel  interoperability*  and 
inveatigationa  of  aupport  concepta  and  environmenta.  Theae  atudiea 
are  uaually  part  of  the  Concept  Exploration  and  Demona ''.ration  and 
Validation  life  cycle  phaaea.  The  deaign  guidelinea  which  reault 
range  from  apacification  of  language  and  operational  computer  of 
choice  to  a  working  prototype  of  the  complete  ayatam. 

GLOSSARY; 


RESPONSE  INSTRUCTIONS; 


RESPONSE  RATIONALE; 


:3XP’,£7ELY  rrSAGRES.  »  !..2.3.4.5,S  »  'TChPLETELV  AGREED 


QUESTION  DATA  SHEET 


# 

QuMtion  Numbar  SPNCDH)  -  002 

SUSSIIS^l  atandarda  for  aoftwara  daalgn  raqulrad  by  tha 

procuramant  activity,  ara  adaquata. 


ACTIVITY <S) ;  Procuramant 

EXPLANATION  t  Tha  atandarda  minimally  includa  davalopmant  contractor 
aoftwara  davalopmant  ragulationa  much  aa  DoD'>ST0~‘2167f  DoD-ST0-2168» 
HIL-STD-483A,  HIL-STD-4S0A,  and  HIL-STD-1S21B.  Ganarally  tha 
RFP/CDRL/SOW  will  indicata.  minimal  aoftwara  daaign  atandarda  and  tha 
ralatad  Data  Itam  Daariptiona  will  indicata  tha  format  and  contant  of 
tha  raaulting  daaign  apacificationa. 


GLOSSARY; 


RgSPONSE  INSTRUCTIONS; 


RgSPONgg  RATIpNA^g; 


RESPO.VSE  SCORE: 


'T.'tPLETTLV  AGI^EE; 


QUESTION  DATA  SHEET 


1 


QuMtlon  Numbar  SPH<DH>  -  0Q3 


gussiisiii  Th«  ao£bwar*  dasign  atathodology  us^d  by  th*  davvlopmsnt. 
contractor  la  adaquata. 


ftynVITY<?), ;  Oavalopnant  Contractor 


?SS?CM3S  I^fSTPUCT:CNS : 


PSSP0N3S  RATIONALE: 


3§s2SL’53-SCORSj. 


A1-S9 


gypLANATIQW  i  Typical  daalgn  aathodologiaa  Includa  approaehaa  for  Ufa 
cycla  avanta,  paraonnal  allocation  by  function,  aupport  raaourca 
aanagaaant,  and  achadula/budgat  analyaaa.  Tha  lifa  cycla  approach 
might  ba  top  down,  itaratlva  rafinaaant,  watarfall,  prototype,  or 
aoaa  combination.  Tha  paraonnal  allocation  by  function  mathod  may 
praacriba  uaa  of  daaign**only,  coda-only,  intagrata-only,  taat-only, 
managamant-only  paraonnal.  Or,  it  may  praacriba  paraonnal  groupa 
which  handle  aoma  combination  of  thaaa  functiona.  Support  raaourca 
managamant  could  includa  apacifieation  of  tha  raaourcaa  auch  aa 
raquiramanta  analyaia  tool,  atructurad  analyaia  daaign  tool, 
automatad  toola  for  apacifieation  generation  and  tranalation  to  PDL 
and  Implamantad  aourca  coda.  Simulatora  for  daaign  apaeificationa  and 
uaa  of  formal  verification  languagaa  are  other  poaalbla  aapacta  of 
daaign  methodology.  Tha  uaa  of  tachniquaa  auch  aa  hierarchy  diagrama, 
KZPOa,  N  by  N  Charta,  data  flow  diagrama,  and  ao  forth  are  important 
for  rapraaanting  tha  daaign  and  are  tha  foundation  of  apacific  daaign 
aiathoda. 


i 


QUESTION  DATA  SHEET 


QuMtlon  Nimb«r  SPN.COK>  -  004 

QUESTION;  Th«  dvslgn  standard*  and  mathoda  adoptad  for  uaa  by  tha 
oparation  aupport  activity  during  poat  daployaant  aoftwara  support 
ara  adaquata. 

•  Oparation  Support 

*2?SWlfAII2ILl  Tha  oparation  aupport  activity  should  adopt  daaign 
standards  and  aathoda  conaiatant  with  intarnal  aits  product  aupport 
standards  and  also  conaiatant  with  tha  daaign  standards  and  mathoda 
uaad  by  tha  davalopmant  contractor.  Tha  CRLCMP  (CHZSP»  0/S  CN?> 
should  includa  information  concarning  tha  daaign  mathoda  salactad. 


O-OSSASV: 


^  SI 


aSgPONgS.  RATIONALE; 


SCORE; 

’TcONPllTlLY'DISAGRii' 


»  1,2,3,4,5,S  »  COMPLETELY  ACRES) 


Al-60 


QUESTION  DATA  SHEET 


Question  Nuatbar  SPN(OK>  -  OOS 

?!7ESTI0N;  Th«  System  D«algn  R«vi«w  process  hss  been  sdsquste. 


ACTIVITY <5^ :  Procursasn^  snd  Operation  Support 

^PLANATIQN ;  The  objective  of  the  System  Design  Review  (SDR)  Is  to 
formslly  essess  the  el located  system  requirements  before  proceeding 
with  the  preliminary  design  of  the  computer  hardware  and  software 
conf iguratlon  items.  The  SDR  should  include  review  of  the  detailed 
system -level  design  and  the  allocation  of  system  functions  to 
individual  hardware  and  software  configuration  items.  The  SDR  should 
include  evaluation  of  the  optimisation,  traceability,  completeness, 
nnd  ri'i!:3  -associated  with  the  allocated  technical  requirements.  A 
succer.sful  SDR  will  be  predicated  on  the  determination  that  the 
Sy^tan  Specif ication  is  an  adequate  basis  for  developing  computer 
hardv-Are  and  software  configuration  items. 

''-IDSSARV : 


Al-61 


QUESTION  DATA  SHEET 


Qu«stlon  Nuab«r  SPM<D!f) 


^tT"ST*ON;  ?>.•  3oftwar«  r*quir«m«nt«  appaar  to  ba  raaaonabla. 


ACTIVITY <3 ) ;  Procuraraant  and  Oparation  Support 

aoftwara  raquiramanta  ara  initially  darivad  at  tha 
functional~‘Iaval  from  procuramant  doeumanta  auch  aa  tha  Stataaant  of 
Naad>  Prograa  Kanagamant  Oiractivaf  Program  Hanagamant  Plan,  and 
avantually  tha  Syatam/Sagmant  Spacification  (A**apae)  .  Tha  aoftwara 
raquiramanta  auat  ba  an  intagrabla,  wall*daflnad  part  of  tha  ovarall 
myatan  raquiramanta,  and  it  muat  ba  claar  what  tha  ralationahip  ia 
among  tha  aoftwara  raquiramanta  and  tha  aystam  miaaion  functiona. 
'Jhathor  tha  aoftwara  raquiramanta  ara  raaaonabla  dapanda  upon:  tha 

total  nunbar  of  raquiramanta;  tha  tachnology  nacasaary  to  implamant 
th-»  raquiramanta  and  aaaociatad  functionality  in  aoftwara;  tha 
srha'iula  and  budgat  for  davalopaant  and  support;  and  tha 
a.'.vironnantal  considarationa  of  aoftwara  parsonnal  akilla,  intarfaca 
raquiramanta  to  tha  ayatam,  parallal  hardwara/aoftwara  davalopmant 
raquiranantJ,  and  tha  ayatam* a  functional  miaaion. 

y.os3Aav: 

A  functional  naad  which  ia  daacribad  in  tha 
ayatam  documantation  and  allocatad  to  aoftwara  for  implamantation . 


satiomals: 


^ — CYrLSTiLY  rT?AG?SS  * 


l,2,3p4,5,6  «  CCXPLSTSLY  AGRSS) 


QUESTIOM  DATA  SHSS 


CuMtlon  Number  SPN(ON)  -  007 

:  Tho  nurber  of  software  requirements  which  cannot  be  traced 
to  an  end  iton  product  is  nininal. 


ACTIVITY  :  All 


Procurenent  ia  responsible  for  the  initial 
identification  and  partial  allocation  of  software  requirements  to 
functional  end  item  components  in  the  System/Segnent  Specification 
^Functional  Sasaline) .  The  operation  support  activity  has  a 
responsibility  to  assist  in  the  definition  of  this  Functional 
Saseline  fros  the  user  and  supporter  viewpoints.  The  development 
contractor  activity  is  responsible  for  continuing  the  software 
.’.ocat  l.rn  process  from  the  high  level  functional  description 
cor.pl'jtely  through  to  the  low- level  software  modules  and  routines.  It 
••■shculd  be  pcssible  to  trace  each  software  requirement  all  the  way 
i.'vr.  to  thA=f  set  of  nodules  and  routines  which  implemsnt  ths 
r'icuiren-nt#  and  to  the  specific  teat  suite  which  verifies  end 
valid«ttes  the  requirement.  All  requirements  should  be  traceable  to 
the  CSC!  level  at  the  conclusion  of  preliminary  design  phase.  See 
AFSC?  SC0-4S  for  more  information. 


SiCSgAFYi 

An  implemented  unit  which  is  not  dscospoeed  further  for 
purpeteo  of  identificst itn.  For  procurement  purpesss,  ths  usual  and 
ti-  is  a  ?f?I.  F-tr  levelopnent  rentrtetor  purposes  (and  related 

ar.i  ir.tsgraticn  testing),  the  ond  item  may  ba  the 
r.i  item,  to  consider  will  also  depend  upon  the  software 
ne. 


-  -  "j-  • , 


routine. 
-  -  - 


-*■ 


Zener  si  Zuideline.  For  acme  eyetems  9C*:  requirements  traceability 
will  be  s  low  ric!';,  for  other  systems  it  may  be  a  high  risk.  Fuzzy 
indicators  era  if  15n  to  2C?:  of  the  requirementa  are  not  traceable. 


then  the  scftwa 

ire 

development 

is 

a 

medium 

to  high  risk  of 

cerpron ..  j  ing  the 
guidelines  is: 

den 

ired  mission 

goali 

1. 

A  more  specific 
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OUSSTION  DATA  SHEET 


QuMtion  Xuabar  SPIKDH) 


00 


\  3c5tw«r«  r^qulrvuanta  which  cannot  ba  taatad 

«ra  stinlnal. 


All 

SXPl  A^ATIt?^  i  In  tha  darivation  of  raquiraaanta  it  la  tha 
raaponaibility  of  all  actlvltiaa  to  apaeify#  at  tha  approprlata  laval 
of  apacif ication,  aoftwara  raquiraaanta  in  a  way  auch  that  taata  can 
ba  dafinad  to  varlfy  and  validate  that  tha  aoft<*ara  raquiraaanta  hava 
baan  mat.  Tha  raquiraaanta  aay  includa  a  tolaranea  range  of  poaalbla 
outcomaa#  a  alnlaua/aaxiaua  abaoluta  valua#  a  aubjaetiva  rating  of  a 
faatura«  or  a  donain/ranga  of  hardwara  and  aoftwara  outeonaa. 
Although  it  la  not  naeaaaary  to  apaclfy  taat  criteria  to  know  whathar 
a  acftvsra  raquiramant  ia  taatabla#  It  la  tha  beat  way  to  maka  aura 
thara  ia  no  miaintarpratation  of  whathar  a  aoftwara  raquiraaant  haa 
paaaad  or  fallad  a'  taat.  Saa  AF8CP  800-4A  for  aora  inforaatlon.  - 


Canaral  Guldalina.  For  aoaa  ayataaa  90k  raquiraaanta  taatabllliv^ 
will  ba  a  low  rlak#  for  other  ayataaa  It  aay  ba  a  high  riak.  FuzzV 
indlcatora  ara  If  19k  to  20k  of  tha  raquiraaanta  ara  not  taatabla. 
than  tha  aoftwara  davalopaant  ia  a  aadiua  to  high  riak  of 
eoaproaiaing  tha  daairad  aiaaion  goala.  A  aora  apacific  aat  of 
guldallnaa  ia: 


F/i; 

E/2: 

0/3: 

C/4: 

B/9: 

A/6: 


Ok  to  90k  raquiraaanta  hava  taatabla  apacificationa 
90k  to  60k  raquiraaanta  hava  taatabla  apacificationa 
60k  to  70k  raquiraaanta  hava  taatabla  apacificationa 
70k  to  60k  raquiraaanta  hava  taatabla  apacificationa 
60k  to  90k  raquiraaanta  hava  taatabla  apacificationa 
9Ck  to  100k  raquiraaanta  hava  taatabla  apacificationa 


RggPpMSg  RATIONALE: 


BKESaSi-SSQSSi _ 

(COMPLETELY  DISAGREE  >  1,2, 3,4, 9, 6  -  COMPLETELY  AGREE) 


Al-64 


QUESTION  DATA  SHEET 


QuMtlon  Number  SPH(DH>  ■*  009 

9SISSIIQN1  The  profile  of  ehengee  to  aoftwere  requlremente  .  is 
reeeoneble. 


^CryviTY iS2l  Procurement  end  Development  Contreetor 

y t  number  of  change  eetiona  (e.g.*  ECPa>  which  impact 
the  aoftwere  requiramenta*  the  eeverity /criticality  of  the  changes, 
and  the  number  of  such  changes  opened/cloaed  over  e  given  time  period 
determine  what  the  change  profile  ia.  This  profile  will  generally 
have  a  higher  number  of  changes  early  in  tha  development,  decreasing 
with  occasional  upward  spikes  to  very  few  changes  near  the  end  of 
development.  Too  many  changes  which  impact  requirements,  changes 
which  ere  extremely  severe,  changes  which  are  open  for  a  long  time, 
and  erratic  increases  and  decreeaem  in  the  unresolved  actions  are 
indicative  of  potential  problems.  Change  requests  result  from  action 
items  which  are  derived  during  informal  and  formal  project  reviews. 
See  AFSCP  S00-4A  for  more  information. 

glossary; 


R§SP0M3|_JNSTRUCTigjiS: 


RggPONSE .RATIONALE; 


A1-6S 


QUESTION  DATA  SHEET 


QuMtlon  Nuab«r  SPM<DM)  •  010 

OUESTIQjil  Th«  proflla  of  unrosolvod  softworo  rovisw  action  itona  la 
raaaonabla. 


;  Procuranant  and  Oparatlon  Support 

RCeUWATIQN;  Unraaolvad  Cop  an)  aoftwara  ravlaw  action  itaaa  raault 
fron  infornal  and  formal  aoftwara  daalgn  ravlawa.  Unraaolvad  action 
itaaa  ara  axpactad  to  apika  upward  at  aaeh  raviaw  and  than  axhibit 
axponantially  dacraaaing  behaviour  •  Pregrama  that  iaaua  claarly 
written  apaclf  icationa  will  axparlanca  apikaa  that  ara  lower. 
Programa  with  good  coaaunicatln  will  have  a  highar  rata  of 
axponantlal  dacay.  The  count  of  unraaolvad  aoftwara  raviaw  action 
Itama  nuat  ba  maintained  by  tha  program  offiea  aa  wall  aa  the 
davalopmant  contractor.  Saa  AFSCP  800-4A  for  mora  Information. 


PSfPgNSE_INSTSyCTI2NSi 


RgyPgN?^  RATI0NA^.E; 


RESPOMSg^SCORgi 
«««•  •• 

i  ^  im. 


o 


■iGnEH) 


QUESTION  DATA  SHEET 


QuMtlon  Nu»b«r  SPHCOH)  *  Oil 

QUESTION;  Thtt  davalopmant  contractor  raqulraaanta  analyala  procaaa 

has  baan  adaquata. 


ftSTSyiTY<S)  ;  Davalopaant  Contractor 

;  A  coaplata  aat  of  functional  and  parforaanca 
raqulraaanta  auat  ba  aatabliahad  for  aaeh  CSCZ.  Tha  raqulraaanta 
analyala  accoapllahad  during  tha  Danonatratlon  and  Validation  phaaa» 
and  aubaaquant  raqulraaanta  validation*  auat  eontinua  during  tha  Full 
Seals  Davalopaant  phaaa  to  coaplataly  daflna  tha  raqulraaanta. 
Zntarfaca  raqulraaanta  auat  ba  daflnad  batwaan  CSCZa  and  HWCIa.  All 
adaptations  naadad  to  accoaaodata  dlffarant  uaar  sltaa  auat  ba 
Idantlflad.  Raqulraaanta  analyala  auat  avaluata  raqulraaanta  for 
coaplatanaaa*  consiatancy*  adaquacy*  taatablllty*  undaratandablllty* 
and  aupportablllty .  As  alaalon  naada  ehanga*  additional  analyaaa  nay 
ba  raqulrad  to  datarnina  tha  Impact  on  aoftwara  raqulraaanta. 


RSgPQWSE  RATIONALE; 


R^POHSS  SCORE; _ 

*  *,.^.3.4. 3.3  »  rCMPLITILY  AGREE: 
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QUESTION  DATA  SHEET 


QuMtion  Ntnibsr  SPH(DH)  -  012 

SUSSXI&lil  dvvalopmant  contractor  top  lovol  dMign  procosa  has 
baan  adaquata. 


*,  O«valopmant  Contractor 


*  A  modular  top*>laval  aoftwara  daalga  abould  ba  davalopad 
from  tha  aoftwara  raquiramanta .  Tha  prallalnary  daaign  procaaa  ahould 
conaldar  varioua  daalgn  altarnativaa*  analytical  raaulta»  trada-off 
atudlaa»  and  capability  to  accommodata  changa.  Tha  daaign  ahould 
Idantlfy  computer  aoftwara  componanta  (CSCa>  and  dafina  tha  data 
Intarfaeaa,  control  flow,  and  raaourca  budgata  for  aamory  and 
axacution  tima  at  tha  CSC  laval.  Functional  aoft«#ara  raquiramanta 
ahould  ba  aaalgnad  to  CSCa  of  tha  top-laval  daalgn.  An  Initial  data 
baaa  daalgn  ahould  dafina  atructura  and  organization  of  tha  data 
baaa.  Tha  daaign  of  formal  and  Informal  taata  ahould  ba  davalopad  and 
documantad  in  tha  aoftwara  plan  for  taatlng  eoapllanca  of  aaeh  CSCI 
with  aach  appllcabla  aoftwara  and  Intarfaca  raqulramant.  Tha 
prallalnary  daalgn  procaaa  culminataa  with  tha  Prallalnary  Daalgn 
Ravlaw  conducted  by  both  proeuramant  and  tha  davalopaant  contractor 


actlvitlaa,  and  monltorad  by  tha  operation  aupport  activity. 


RESPONSE  INSTRUCTIpNS; 


RESPONSE, RATIONALE; 


5g2225i5i-5S2SSi _ 


>  w  a  w  ^  • 


»  CChPLITELV  agree; 


QUESTION  DATA  SHEET 

QuMtlon  Nuxbar  SPHCOH)  -  013 

QUESTION ;  Th*  davalopmsnt  eontracbor  datallad  daaign  procaaa  haa  baan 
adaquata . 

ftCTiyiIYi82I  Davalopaant  Contractor 

datallad  aoftwara  daaign  procaaa  ahould  raflna  tha 
C^a  of  tha  top-laval  aoftwara  daaign  to  aueeaaalvaly  lowar-laval 
daaign  alaaanta  until#  at  tha  lowaat  laval#  thay  apacify  individual 
unita  to  ba  davalopad.  Tha  datailad  daaign  ahould  dafina  all 
information  raquirad  for  coding  thaaa  unita#  including  control  logic# 
algor ithaa#  data#  accuracy#  and  timing.  For  any  intarfacaa  with  othar 
CSCIa  or  HWCIa#  datailad  intarfaca  daaign  ahould  praciaaly  dafina 
data  formata#  data  flow#  and  timing  conatrainta  in  aufficiant  datail 
for  coding  data  atructuraa  and  control  routinaa.  Tha  data  baaa  daaign 
ahould  ba  dafinad#  Including  ita  conatituant  itaaa#  fialda#  racorda# 
and  filaa.  Tha  datailad  aoftwara  taat  daaign  ahould  dafina  tha 
nathoda  and  critaria  of  conducting  tha  individual  taata  pravioualy 
idantifiad  in  tha  Softwara  Taat  Plan.  Each  taat  caaa  ahould  ba 
daaignad  in  tarma  of  inputa#  ampactad  raaulta#  and  avaluation 
critaria  <a.g.#  paaa/fail).  Tha  taat  daacriptiona  form  tha  baaia  for 
aubaaquant  davalopaant  of  taat  procaduraa.  paaeriptipna  of  formal 
taata  ahould  raquira  procuraaant  activity  approval.  Tha  datailad 
daaign  procaaa  culminataa  with  tha  Critical  Daaign  Raviaw  conductad 
by  both  tha  procuramant  and  tha  davalopaant  contractor  activitiaa# 
and  monitorad  by  tha  operation  aupport  activity. 

RgSPpN?E  INgTRUCT^ON?: 


I*:* 


RESPONSE  RATIONAI^E; 


Kil 


•M 


iM 


RE3P0NSE_SCQREX 


QUESTION  DATA  SHEET 


QuMtion  Nu«b«r  SPH(DM) 


OUESTIOW;  Th«  daslgn  co»pl«tlon  of  CSCZa  rolativa  to  tho  aoftwaro 
Ufa  cyela  davalopaant  achadula  haa  baan  raaaonabla. 


4SXI^IXXiSll  Davalopaant  Contractor 

EffiWfiAIISfli  Tha  rata  at  which  a  davalopaant  contractor  coaplataa 
CSCI  daalgna  aay  vary  dapanding  upon  tha  aoftwara  davalopaant  aathod 
aalactad.  A  prototypa  aathod  aay  allow  for  aoaa  CSCZa  to  ba 
coaplataly  codad  bafora  othar  CSCZa  ara  avan  daalgnad.  Tha  daalgn 
eoaplatlon  crltaria  aay  thua  ba  aoaawhat  aubsactlva  and  ahould  ba 
tallorad  to  tha  particular  ayataa.  Ganarally^  tha  raaponaa  guldalinaa 
ahould  raflact  adaquata  eonaidarationa  of  ayataa  Ufa  cycla  phaaaa 
and  aoftwara  davalopaant  aathodology  baing  uaad.  Saa  AFSCP  800-4d  for 
aora  information. 

Bff^gii-iyoaplfation.  CSCZ  haa  aatiaf actor lly  eoaplatad  ita  data! lad 
daaign  apacif  ication  <C-apac>  • 

ffff tgn .  Pf f ^ciaiyqv j  (Nuabar  of  CSCZa  plannad  for  daaign  eoaplatlon 
ainua  tha  nuabar  of  CSCZa  actually  daaigmMI)  divided  by  tha  (nuabar 
of  CSCZa  plannad  for  daaign  eoaplatlon)  all  tiraaa  100.  _ 


F/i:  sox  to  lOOx  daaign  dafieiancy 
E/2:  40X  to  SOX  daaign  dafieiancy 
D/3:  30X  to  40X  daaign  dafieiancy 
C/4:  20X  to  30X  daaign  dafieiancy 
3/S:  ipx  to  20X  daaign  dafieiancy 
A/6:  OX  to  lOX  daaign  dafieiancy 

RESP0NSg„  R  ATigNAyg : 


3§§£21!§5_sccre2 _ 

“<C0MPLiTELY"DlSAGREE~'’l,2,3,4,S,6  ■  COMPLETELY  AGREE) 


QUESTION  DATA  SHEET 


QuMtion  Nuabcr  SPH(OH>  -  019 

9UEST^QN :  Th«  dav«lop»«nt  contractor  monitor  of  tho  subcontractor 
software  design  process  has  bean  adequate. 


ACTIVITY <S) ;  Davelopsant  Contractor 

B^P^j^yATIQN ;  Any  subcontractora  to  the  prise  developaent  contractor 
who  have  software  developaent  responsibilities  should  be  required  to 
apply  design  standards  and  methods  consistent  with  the  prime 
contractor's  required'  standards  and  methods.  The  prime  contractor  has 
ultimate  responsibility  to  the  procurement  activity  for  delivery  of 
quality  software*  hence  subcontractor  efforts  in  this  ares  must  be 
carefully  monitored  and  reviewed*  much  es  the  procurement  activity 
monitors  and  reviews  the  development  contractor  software  development 
effort. 

GLOgSAHY! 


A/6:  There  are  no  subcontractors  with  software  development 
responsibilities 


RESPONSE^RATlONAlgg: 

I 


RESPONSE  SCORE: 


‘LZTEL?  .iCncE: 
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QUESTION  DATA  SHEET 


vsy 

QuMtion  Muab«r  SPH(DH)  -  016 

QUESTION ;  Th«  d««lgn  •pacifications  for  th«  softwars  products  contsln 
sdsqusts  informstlon  to  Isplssont  tho  softwsrs  with  ths  required 
functionality  end  within  the  schedule  snd  budget  requirements. 

^SnyilYiSli  All 

gyp^^NAT^ON :  The  design  specif  lest  Ions  Include  the  System/Segment 
Specification »  the  top-level  design  specif  icetlon#  the  detslled 
design  specifications#  Interface  specifications#  data  bsse  design 
specifications#  snd  the  test  design  specifications.  The  design 
specifications  should  sdequately  capture  the  transformation  of 
requirements  Into  a  paper  representation  of  the  software  solution# 
•ufficlantly  precise  to  directly  Implement  the  software. 


G^CSSARY^ 


RE?PpN?E  RATIONALE! 


5sS29ii§S.SS3SSl 

”<C0MPLETiLY"DISACRii”«“l,2,3,4#5,6  ■  COMPLETELY  AGREE) 
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QUESTION  DATA  SHEET 

QuMtlon  Nunb«r  SPH(DH)  -  017 

QUESTION;  Th«  operation  support  eoneapt  for  design  of  software 
revisions  during  post  deploynent  software  support  is  adequate. 


ACTIVITY <S) ;  Operation  Support 

operation  support  concept  should  include  design 
methods,  top-level  end  detailed  design  process  approaches,  and  test 
design  methods.  This  concept  should  be  consistent  with  the  concept 
used  by  the  development  contractor. 


e 


RESPONSE  instructions; 


RESPONSE  RATIONALE: 


_ 

”<c5n?LSTELY“5 IS AGREE  -  1,2,3,4,S,6  -.COMPLETELY  AGREE) 
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I 

I 


QUESTION  DATA  SHEET 


QuMtion  Number  SPH(OH)  -  Old 


SsiSSIISil  operation  support  concept  for  design  review  during  poet 
deployment  software  support  is  adequate. 


Operation  Support 

operation  support  concept  should  include  preliminery 
and  detailed  design  reviews  as  part  of  the  software  block  release 
process.  The  reviews  should  be  consistent  with  those  used  during  Full 
Scale  Development,  probebly  at  a  somewhat  reduced  scope, 
proportionate  to  the  extensiveness  of  the  changes  in  the  block 
release  and  how  much  the  software  design  has  been  affected  by  the 
changes . 


SLOSSASY ; 


NST^^’JCTIONS; 


PS3P0N3S  HATIQ?fAVE! 


SOFTWARE  PROJECT  WAlfACERENT  IMPLEMEMTATIOM  METHODS 


Th«  quastlons  SPK(ZH>-001  through  SPH<ZH)*‘016  oddross  «d«qu«cy 
of  software  project  sansgsasnt  isplsaantatlon  ssthods  for  ths 
procuromsnt*  dsvslopssnt  contractor »  and  operation  support 
actlvltlas.  Project  aanageaent  lapleeentatlon  aathoda  are 
eatabliahad  so  that  the  design  specifications  of  a  project  can  be 
more  efficiently  transcribed  into  an  implemented  product.  The 
implementation  methods  include  standards#  conventions#  regulations# 
directives#  software  language#  review  methods#  low  level 
representation  techniques  and  methodologies#  automated  implementation 
aids#  and  so  forth. 

There  are  generally  t%#o  aspects  of  an  iaplemantation  method 
which  need  to  be  evaluated.  First#  does  the  implementation  method 
facilitate  production  of  high  quality  software  within  limited 
available  resources?  Second#  can  the  implementation  method  be 
transitioned  to  the  software  support  activity  for  the  evolution  of 
the  software  products  during  post  deployment  support? 

The  procurement  activity  is  responsible  for  assuring  that 
adequate  implementation  methods  (e.g.#  coding  standards#  dask  cheek 
procedures)  have  been  required  ‘  through  the  PMP#  CDRL#  SOW#  and  any 
other  procurement  documents.  The  procurement  activity  is  also 
responsible  for  understanding  the  nature  and  importance  of  the 
development  contractor  implementation  methods  and  the  effseta  which 
software  implementation  tradeoffs  might  have  upon  the  desired  system. 
The  procurement  activity  should  have  its  own  methods  of  assessing 
whether  software  design  and  the  implementation  of  that  design  are 
consistent#  and  whether  the  implementation  is  an  adequate 
reqpreaentation  of  the  design. 

The  development  contractor  activity  is  responsible  for 
establishing  and  using  implementation  standards  appropriate  for  the 
software  being  developed  so  that  an  operationally  effective  and 
supportable  system'  is  produced.  Implementation  representation 
techniques  and  automated  implementation  aids  should  be  appropriate 
for  coding  and  integrating  the  software  in  an  efficient  manner  and 
for  transitioning  the  techniques  and  aids  to  the  operation  support 
activity. 

The  operation  support  activity  is  responsible  for  making  mure 
the  development  contractor  has  requirements  to  use  implementation 
methods  which  can  be  effectively  used  during  post  deployment  support. 
Such  methods  may  require  training  for  the- support  activity  personnel. 
The  operstion  support  activity  also  has  the  responsibility  to  assure 
that  the  software  development  environment  is  delivered  with  all  the 
necessary  Impementation  aids  to  effectively  accomplish  software 
support. 


Al-75 


QUESTION  DATA  SHEET 


QuMtlon  NuBbttr  SPM<IN>  **  001 

QUESTION;  Th«  proeur«»«nt  •ctivity  h«a  adaquataly  aonltorad  tha 
iaplaaantation  of  tha  aoftwara  daalgn  apaclfleationa. 


ACTIVITY <211  Procuraaant 

g}(pLANATIONt  Tha  tiaa  gap  bati^an  tha  and  of  tha  major  datallad 
daai$}n  phaaa  (Critical  Oaaign  Raviaw>  and  tha  baginning  of  ayatam 
intagration  taating  aa  aignalad  by  tha  Taat  Raadinaaa  Raviaw  can  ba 
aignificant.  It  ia  nacaaaary  for  tha  procuraaant  activity  to 
carafully  monitor  davalopmant  contractor  iaplamantation  prograaa 
through  atatua  raporta,  informal  raviawa,  aita  viaita,  interim 
damonatrationa,  and  tha  raquirad  aoftwara  baaalina  change  procaaa. 


PV9^?ARY; 


RggPONSg  INgTRUCTIONg ; 


RESPONSE  RAT^ONALg; 


W-i.vJimyiiBugMw^iAnnj 


QUESTION  DATA  SHEET 

QuMtion  Nuab«r  SPMdH)  -  002 

QUESTION;  Th«  procurMMt  tMt  org«nlz«tion  lnt«rf«e«  with  th« 
davsiopiiiant  contractor  la  adaquata  anouqh  to  aaaxira  aliccaaa  of  tha 
ayataa  Intagratlon  taata. 

^gJJVITY<S> ;  Procuraaant  and  Davalopaant  Contractor 

SyPWAWAIIflS.?  primary  taat  organlzatlona  ara  tha  DT6E  agancy,  tha 
OTAE  agancy*  and  poaalbly  an  IVAV  organization.  Tha  ayatam 
Intagratlon  taat  auecaaa  la  dlractly  dapandant  upon  tha 
Implanantatlon  prograaa  of  tha  davalopaant  contractor.  Tha 
oparatlonal  taat  procaaa  alao  haa  auch  dapandancy.  Tha  IV6V  (If  any) 
will  ganarally  provlda  for  aupplamantary  taatlng  which  provldaa 
graatar  aaauranca  that  tha  Implanantatlon  la  all  right  and/or 
Idantlflaa  problan  araaa  which  daeraaaa  aaauranca  and  langthan  tha 
Implanantatlon  phaaa. 


S^ggAgV! 


RS^poNss  ;N?7aucT;oN? ; 


RESPONSE  _gATT9NAIff ; 


(  :0?*?L2r'ilV  DISAGREE  ■  1.2. 3. 4. 3, 5  »  COMPl-STSL.  iGSEE) 
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QUESTION  DATA  SHEET 


QuMtlon  Nuabar  8PH(IK>  ~  003 

QUESTION;.  Tha  oparation  aupport  activity  haa  baan  aetivaly  involvad 
with  tha  davalopaant  contractor 'a  aoftwara  iaplaaantation  in  ordar  to 
laarn  tha  aoftwara  prior  to  officially  accepting  aoftwara  aupport 
raaponaibil'ity . 

ACTIVIXY^S) ;  Operation  Support 

;  All  through  tha  full  acala  davalopaant  tha  oparation 
aupport  activity  ahould  aonitor  tha  prograaa  of  tha  davalopaant 
contractor 'a  aoftwara  davalopaant.  During  tha  iaplaaantation  phaaa 
(lattar  part)  aoaa  key  oparation  aupport  activity  paraonnal  ahould 
bagin  to  actively  laarn  tha  aoftwara  daaign#  iaplaaantation » 
intagration.  and  taat.  Thia  aay  taka  tha  fora  of  foraal  eouraa 
training  and  handa  on  aoftwara  aodification  to  inforaal  obaarvationa 
of  tha  davalopaant  contractor  procaaa. 


SkSSSASYl 


5E2£QS5S-ia2I52SII2S2i 

r/l:  Thera  are  no  plana  for  oparation  aupport  paraonnal  to 

actively  participate  in  tha  aoftwara  aodification  procaaa 
prior  to  PKRT. 


RESPONDS  RATIONAL^: 


<CrN?L2TSI.Y  DISAGREE 


1  '’■943. 


CCKPLETSLY  AGREED 
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QUESTION  DATA  SHEET 


QuMtion  Nu»b«r  SPH(IH>  -  004 

atandsrda  for  softwaro  laplamantatlon  raqulrad  by  tha 
procuraaant  activity  ara  adaquata. 


ACTIVITY *  Procuraaant 

atandarda  ainiaally  includa  davalopaant  contractor 
aoftwara  davalopaant  raqulationa  auch  aa  DoD-STO-'QlS?^  OoD-STD-2168, 
MZL-STD-483A,  MZL-ST0-490A,  and  MIL*STD-1921B.  Canarally  tha 
RFP/CDHL/SOW  will  indicata  ainlaal  aoftwara  iaplaaantation  atandarda 
and  tha  ralatad  Data  Ztaa  Daariptiona  will  indicata  tha  foraat  and 
contant  of  tha  raaulting  iaplaaantation  apacificationa. 


INSTRUqT^OWS; 


RESPONSE  RATIONAL^: 


« 

7"5i3AGaii" 


.2.3,4.3.S 


■»  OOMPLETELV  AGREE? 
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QUESTION  DATA  SHEET 


QuMtion  Nu»b«r  SPM<IH>  ~  00 


OUESTIQN^  Th«  ImplsKantation  methodology  umod  by  tho  dmvmlopmmnt 
contrmetor  ia  adaquata. 


ftyryvixycg) ;  Davalopaant  Contractor 


;  Typical  inplaimantatlon  aathodologiaa  include  approaehaa 
for  life  cycle  avanta*  paraonnal  allocation  by  function,  aupport 
raaotirca  nanagamant,  and  achadula/budgat  analyaaa.  The  life  cycle 
approach  might  be  top  down,  iterative  rafinamant,  waterfall# 
prototype#  or  aoma  combination.  The  paraonnal  allocation  by  function 
method  may  praacriba  uaa  of  daaign-only#  coda-only#  integrate-only# 
taat-only#  managamant-only  paraonnal.  Or#  it  may  praacriba  paraonnal 
groupa  which  handle  aoma  combination  of  thaaa  functiona.  Support 
raaourca  managamant  could  include  apaeif Ication  of  the  raaoureaa  auch 
aa  raquiramanta  analyaia  tool#  atructurad  analyaia  daaign  tool# 
automated  toola  for  apaeif  ication  generation  and  tranalation  to  PDL 
and  imp lamented  aourea  coda.  Simulatora  for  daaign  apaeif ieationa  and 
uaa  of  formal  verification  languagaa  are  other  poaaibla  aapacta  of 
implamantation  methodology.  The  uaa  of  taehniquaa  auch  aa  coda 
walkthrougha#  aimulation#  aymbolie  debug#  atatie  coda  analyaia# 
automated  teat  eaaa  ganaratora#  ragraaaion  taating  arc  are 
foundation  of  eoda/unit  taat/intagration  implamantation  aathoda. 


3s222i!SS-Il!5IS2SII232i 


RggPONSE  ^  RAT^gWALE ; 


RESPONSE  SqpRE;  _ 
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♦ 


QUESTION  DATA  SHEET 


QuMtlon  Numbar  SPM(ZH)  -  006 

QUESTION:  Tha  implaKantatlon  atandarda  and  aathoda  adoptad  for  uaa  by 
tha  oparation  aupport  activity  during  poat  daployaant  aoftwara 
aupport  ara  adaquata. 

*  Oparatlon  Support 

oparatlon  aupport  activity  ahould  adopt 
laplaaantatlon  atandarda  and  aathoda  eonalatant  with  Internal  alta 
product  aupport  atandarda  and  alao  eonalatant  with  tha  lapJ aaantatlon 
atandarda  and  aathoda  uaad  by  tha  davalopaant  contractor.  Tha  CRLCMP 
(CRISP,  0/S  CMP)  ahould  Include  Inforaatlon  concerning  tha 
laplaaantatlon  aathoda  aalactad. 


GLOSSARY: 


Rg^poNSS  instructions: 


SSS£Sl!SS.BmSHAWSl 


RESPONSE  SCORE; _ _ 

*  completely  agree; 


Al-81 


QUESTION  DATA  SHEET 


QuMtlon  Nuab^r  SPHCZM)  ~  007 

qyESTIQNj.  -Th«  davalopaant  contractor  monitor  o£  subcontractor 
software  iaplsaantatlon  process  has  been  sdequate. 

ACTIVITY (g).!  Oevelopsent  Contrector 

EXPLANATION :  Any  subcontrsetors  to  the  prise  development  contractor 
who  have  software  development  responsibilities  should  be  required  to 
apply  implaaentation  standards  end  methods  consistent  with  the  prime 
contractor's  required  standards  and  methods.  Ths  prims  contractor  has 
ultimate  raaponaibility  to  the  procurement  activity  for  delivery  of 
quality  software,  hence  subcontractor  efforts  in  this  area  must  be 
carefully  monitored  and  reviewed,  much  as  the  procurement  activity 
monitors  and  reviews  the  development  contractor  software  development 
effort* 


?g^P9NSg  ;NgTHUCTlgN|g^ 

A/6:  There  are  no  subcontractors  with  software  development 
raaponaibility 


5i§S2*fSE  ^cgRE^ _ 


^  TChPLITILY  AGHSS: 


#  w  •  w  •  ^  v.w  *  <9 


QUESTION  DATA  SHEET 


QuMtlon  Nu»b«r  SPHdH)  -  008 

QUESTION;  Th*  luplanantation  eoaplation  of  CSCZa  h«a  boon  roooonoblo 
rolativo  to  tho  ooftworo  llfo  cyclo  ochodulo. 


ACTIVITY  Dovolopoont  Contractor 

SSBiihmiQn  Tho  rata  at  which  a  dovolopaont  contractor  coaplotoa 
CSCI  iaplomontationa  nay  vary  doponding  upon  tho  ooftworo  dovolopaont 
aothod  aoloctod.  A  prototypo  aothod  nay  allow  for  aoao  CSCIa  to  bo 
coaplotoly  toatod  boforo  othor  CSCZo  aro  ovon  doaignod.  Tho 
inplonontatlon  conplotlon  critorla  nay  thua  bo  aonowhat  aubjoctivo 
and  ahould  bo  tailorod  to  tho  particular  ayaton.  Gonorally*  tho 
roaponao  guldollnoa  ahould  rofloct  adoquato  conaldoratlona  of  ayaton 
llfo  cyclo  phaaoa  and  ooftworo  dovolopaont  nothodology  bolng  uaod. 
Soo  AF3CP  800*48  for  aoro  Inforaotlon. 

GLOSSARY ; 

Inplonontatlon  of  a  C'iCZ  la  conploto 
whan  tho  CSCI  haa  aatlafactorlly  conplotod  Ita  Intagratlon  tooting. 

Inplonontftlqq  ^Doficloncv.^  (Nunbor  of  CSCIa  plannod  for 

Inplonontatlon  conplotlon  nlnua  tho  nunbor  of  CSCIo  actually 
laplonontod)  dAvldod  by  tho  (nunbor  of  CSCI*  plannod  for 

Inplonontatlon  conplotlon)  all  tlnoa  100. 


RESPONSE  INSTRUCTIONS; 

F/i:  SOX  to  lOOX  Inplonontatlon  dofleloncy 
E/2:  40X  to  SOX  Inplonontatlon  dofleloncy 
D/3:  30X  to  40X  Inplonontatlon  dofleloncy 
C/4:  20X  to  30X  Inplonontatlon  dofleloncy 
9/S:  lOx  to  20X  Inplonontatlon  dofleloncy 
A/S:  ox  to  lOX  Inplonontatlon  dofleloncy 

RE^PQNSS  RAT|0NAI|.^: 


Rg^PgNSg  ,?<?0Rg: _ 

*(C0MPLSTELY~0 IS AGREE  -  1.2,3,4,S»6  »  COMPLETELY  AGREE) 


Ai-as 


QUESTION  DATA  SHEET 

QuMtlon  Nuabttr  SPHCIM)  -  009 


QUESTION;  Thtt  procuranant  aoftwar*  projact  mmnmgmmmnt.  support  tool 
•nvironsont  la  adaquata. 


ACTIVITY <S) ;  Procuraaant 

pCPyANAJJQtf^  During  tha  Concapt  Exploration  and  Danonatration  and 
Validation  phaaaa*  tha  nacaaaary  autoaatad  tools  and  proeaduraa  for 
procuraaant  projaet  aanagaaant  Caoftwara  and  hardwara)  should  ba 
idantifiad#  davalopad,  and/or  aequirad.  Tha  tool  anvironaant  should 
allow  for  budgat  and  achadula  aanagaaant*  aisaion  raquiraaants 
tracing*  product  dalivarabla  status  tracking*  configuration 
aanagaaant  changa  status  tracking*  and  aanagaaant  inforaation 
comaunication  capabilitiaa  aaong  participating  organizations.  A 
currant  list  of  raquirad  tools*  function  .of  aach  tool*  data  raquirad* 
and  data  aequirad  should  ba  aaintainad. 


S*»iS2N§S.IN2I52SIIQS!2i 


RgSP0NSg,RAT|0NA;,^; 


riSAGSSS  •  1,2.3.4.5,1s  »  CCKPLETSLY  *GSSS> 


Al-64 


QUCSTZON  DATA  SHEET 


QuMtion  Nuab«r  SPMCIM)  -  010 

pVg^TION;  Th«  dav«lopai«nt  contractor  software  project  nanageaent 
support  tool  envlronaent  is  adequate. 

ACTIVITY <S> :  Development  Contractor 

development  contractor  activity  is  required  to 
provide  certain  management  information  on  the  project  statue  to  the 
procurement  activity.  Automated  project  management  tools  are 
available  to  aaaiat  these  software  project  managekant  functions.  The 
DoD-8TD-2167  and  ite  asaociatad  Data  Item  Descriptions,  aa  called  out 
in  the  CDHL,  have  specific  requirements  for  development  contractor 
'data  collection  and  reporting. 


GLOSSARY : 


3“§E2-l§I.I-!§ISi’SII.2s!§I 

If  no  automated  project  management  tools  are  being  used,  than  th* 
response  should  be  leas  than  C/4. 


RS3P0MSE .RATJpNA^f i 


.RSSPOMSS  SCORE: 


A1-6S 


•  •  •  • 


QUESTION  DATA  SHEET  ^ 

QuMtion  NuKbar  SPM(IH)  -  Oil 

QUESTION ;  Th«  d«v«lopM«nt  contractor  «oftw«r«  configuration 
nanaganant  support  tool  anvironaant  ia  adaquata. 

ACTIVITY <S? ;  Davalopaant  Contractor 

Softwara  configuration  aanagaaant  ia  ona  of  tha  moat 
important  aanagaaant  functions  parforaad  by  tha  davalopaant 
contractor.  Fraquantly  tha  amount  of  information  to  ba  ratainad# 
analyzed#  and  raportad  ia  vary  larga.  Thua#  an  automated  support  tool 
anvironaant  to  aaaiat  this  procaaa  ia  aaaantial.  Such  a  tool  must  ba 
aupplaaantad  with  adaquata  aanagaaant  procaduraa.  Such  a  tool 

anvironaant  must  ba  must  have  tha  capability  to  afficiantly  report  on 
all  software  components  under  configuration  control#  currant  and 
planned  ehangaa  to  those  components#  and  planned  eoaponanta  not 
currently  under  control.  Library  aanagaaant  of  software 

(specif icat ions#  source#  object#  coaaand  language#  load  modules#  teat 
data#  etc.)  and  the  capability  to  automatically  reconstruct  current 
and  previous  versions  of  software  components  ia  required. 


2sS2£'iIix-liSSI2^2IISS3l 

If  no  automated  project  management  tools  are  being  used#  then  the 
response  should  be  less  than  C/4. 


SISSSSH.5AIISlfA&£l 


SsSS3iin:.sss5Si _ 


A1-S6 


QUESTION  DATA  SHEET 


QuMtlon  Nu»b«r  SPHdH)  -  012 

CUESTIOKL  Thtt  davalopmant  contractor  ayatom  aoftwara  tool  anvlronmant 
la  adaquata. 


Davalopnant  Contractor 

SS^EimilQSl  Tha  praciaa  conf iqxjratlon  o£  a  ayataa  davalopaant  tool 
anvlronmant  will  vary  aoaawhat  dapandlng  upon  tha  particular 
application  and  tha  coaplaxlty  of  tha  aoftwara  davalopaant  affort. 


Operating  ayataa,  coapilar,  linker, 
debugger,  data  baaa  nanagar,  aathodology  aupport  tool,  raqulraaanta 
generation  tool,  hoat  ayataa,  and  ao  forth. 


RggPOWSf  INSTRUCTI9N? ; 


5S52Sl!2i-SATigNAi,Ei 


REJPOMSE^SepRE: _ 

“cCOMPlItElVdIsaGREE  -  1,2,3,4,S,6  ■  COMPLETELY  AGREE) 


Al-67 


QUESTION  DATA  SHEET 


0# 


QuMtion  MuBb«r  SPM(ZH) 

9y^SXIQl!l  Th*  dBVBlopBBnt  contractor  application  aoftwara  taat 
anvironaant  la  adaquato. 


ACTIVITIISII  Davalopmant  Contractor 

taat  anvironaant  tool  aat  la  critical  for  thorough 
iinlt  taatlng  and  laboratory  Integration  taatlng.  Tha  aaturlty  of  tha 
aoftwara  alll  largely  dapand  upon  how  coaplataly  tha  aoftwara  can  ba 
taatad. 


OLOSSARY: 

software  banch,  target 
naehina»  laboratory  Integrated  taat  bad*  apaclallzad  taat  davlcaa  and 
Inatruaantatlon*  apaclal  aacurlty  facllltlaa*  alaulatora  and 
anulatora*  and  ao  forth. 


SSSB2HS5.ISSIBUSIISNS1 


RSyqWSE  NATiqNi^I,^; 


(CC:?PL2THLY  DISAGREE  »  1,2. 3. 4, 5. 5  ■  COMPLETELY  AGREE) 


QUESTION  DATA  SHEET 


QuMtlon  Nuab«r  SPH(IM)  *■  014 

OUE^T^QNl  Tha  oparatlon  support  softwsro  support  tool  snvlronmsnt  Is 
sdsqusts. 


6SXiyU2iS2i  Opsrstlon  Support 

6yPLAyATT9?f  ^  softwsrs  support  tool  snvlronasnt  Is  ons  o£  ths 
eritlcsl  factors  for  softwsrs  supportsblllty .  Ths  software  rovisions 
must  bs  dsvslopsd#  configuration  control lsd«  and  tostsd  in  s  ssnnsr 
ainilsr  to  ths  original  Full  Seals  Dsvslopssnt  effort. 


SitSSSARYi 

Support  Tool  Eovironssnt.  Ths  systass  of  ths  Softwsrs 
Support  Rssourcss.  Zneludss  ths  systss  dsvslopssnt  softwsrs  tool 
snvironssnt  and  ths  application  software  test  tool  snvironssnt  as 
required  by  ths  support  snvironssnt. 


5S§SQl!5I-I-lSISliSII2*£§i 


RESPONSE  rationale: 


S52EQS5S-SS0RSi _ 

CCOKPLEtIly'oISAGREE  ■  1,2,3,4,S.6  -  COMPLETELY  AGREE) 


QUESTION  DATA  SHEET 


QuMtlon  Number  SPM(IM)  -  019 

QUESTION  1  Th«  operation  support  concept  for  inpleeentation  of- 
eoftwere  revlalone  during  poet  deployeent  eoftwere  support  is 
edequete . 


tSimilllUl  Operation  Support 

operation  support  concept  should  include  coding 
style  guidelines^  sethods  for  assuring  code  correctneaa  (e.g., 
quality  assurance  setrics*  code  desk  checks)  prior  to  unit  tast*  unit 
and  integration  test  sethods#  and  operational  test  sethods.  The 
concept  should  include  an  overall  software  support  plan  which 
delineates  how  software  releases  are  to  be  prosect  and  configuration 
sanaged.  This  plan  is  an  internal  detailed  specification  derived  from 
the  sore  high  level  CNLCMP  (CRISP#  0/S  CMP). 


RSSSQHSS.INSI5USIISNS1 


5SS£Sl!S5.5mQS&kSl 


_ 

“TcOMPLSTSLY'orSAGRis  ■  1,2. 3. 4. 9. 6  •  COMPLETELY  AGREE) 


Al-90 


QUESTION  DATA  SHEET 


QuMtion  Nu»b«r  SPH(IN)  -  016 

0UE3XI0W  «■  Thtt  operation  support  concept  for  implementation  audita  end 
reviews  during  poet  deployment  software  support  is  adequate. 


ACTIVITY  <S) ;  Operation  Support 

pcpyyMi^T^ON ;  The  review  of  implemented  revisions  has  some  parallel 
with  the  revieva  and  audita  in  the  development  process.  However »  the 
reviews  tend  to  be  lees  extensive.  A  Teat  Reedinese  Review  should  be 
conducted  on  the  revised  eoftwere  baseline  prior  to  operational 
testing  of  each  block  release.  Informal  reviews,  smell  aeele 
functional  and  physical  conf igiaretion  audit/reviews  for  configuration 
baseline  update  integrity,  end  the  teat  readiness  review  constitute 
the  audits  and  reviews  pertinent  to  the  implementation  phase  of  e 
block  release. 


3s5E2!f§S-IS22S2SII9S2i 


S£2£S!!2S.5AII0NAt£i 


# 


S§2E22!2£_2SSSfi 


,  -3 . 4 . 3 . 3 


SOFTWARE  PROJECT  MANAGEMENT  TEST  STRATEGIES 


Th«  quMtlon*  SPN(TS)<-001  through  SPN(TS>‘*020  addroM  adequacy 
o£  aoftwara  project  aenagaaant  teat  atretegiea  for  tha  procurement » 
davalopaant  contractor »  and  oparation  support  ectJLeitlaa.  Software 
taat  atretagiaa  are  aatabllahad  ao  that  the  iaplaaMBtad  product  can 
ba  varlflad  and  valldatad  agelnat  tha  requlreaant  apaclficatione .  The 
taat  atretagiaa  Include  etandarda#  convantiooa^  raguletlona# 
direct ivaa»  taat  languagaa#  verification  and  validat.ion  aathoda# 
taat  generation  and  rapraaantation  teehniquaa  and  aathodologiea* 
autoaatad  taat  aide,  end  ao  forth.  Kay  to  a  raaaooabla  taat  atratagy 
ia  tha  generation  of  a  taat  plan*  taat  deeeriptioM*  test  procaduraa* 
taat  raporta*  demonatration  taata*  configuration  aanegaaent  of  taat 
inforaetion*  and  traneition  of  taat  atratagy  to  tha  operation  support 
activity. 


Thera  are  generally  two  eapacta  of  a  teqt  atratagy  which  need  to 
ba  evaluated.  Firat,  doaa  tha  taat  atratagy  facilitate  production  of 
high  quality  aoftwara  within  liaitad  available  raaourcaa?  Second*  can 
tha  taat  atratagy  ba  uaad  by  tha  software  support  activity  for  tha 
evolution  of  tha  aoftwara  products  during  poet  daployaant  support? 


Tha  procurement  activity  ia  raaponaibla  for  assuring  that 
adequate  taat  aathoda  have  bean  required  or  are  planned  through  tha 
PMP*  CORL*  SOW*  TEMP*  DTAE  and  .  OTAE  taat  plana*  and  any  othaj^ 
procuraaant  doeumanta.  Tha  procuraaant  activity  ia  also  rasponaibl^ 
for  understanding  tha  nature  and  importance  of  tha  davalopaant 
contractor  taat  atretagiaa  and  tha  affacta  which  various  testing 
techniques  eight  have  upon  tha  verification  and  validation  of  the 
daairad  ayatam.  Tha  procuraaant  activity  ia  raaponaibla  for  making 
aura  tha  0T6E*  OTfcE*  ZV6V*  and  davalopaant  contractor  test  atretagiaa 
are  conaiatant  and  are  comp  lament  ary .  Tha  procuraaant  activity  should 
have  its  own  methods  of  aaaaaaing  whether  aoftwara  raquiramants  have 
bean  adequately  verified  and  validated*  and  tha  amount  and  type  of 
tasting  which  ia  atill  required  to  achieve  an  operational  capability. 


Tha  davalopaant  contractor  activity  ia  raaponaibla  for 
establishing  taat  standards  appropriate  for  tha  aoftwara  being 
developed  ao  that  an  operationally  affective  and  aupportabla  ayatam 
ia  produced.  Taat  plana*  techniques*  achadulaa*  and  automated  aide 
should  ba  appropriate  for  thorough  •  taat  of  tha  aoftwara 
implamantation  and  ayatam  integration.  The  taat  techniques*  taat 
caaaa*  and  ta;at  environment  should  ba  designed  for  transition  to  tha 
oparation  support  activity. 

Tha  oparation  support  activity  ia  raaponaiblja  for  making  aura 
tha  davalopaant  contractor  ia  required  to  use  a  taat  strategy  which 
can  ba  aff  actively  uaad  during  poet  daploymant  support.  Tha 
oparation  support  activity  also  has  tha  responsibility  to  aaaura  that 
tha  post  deployment  aoftwara  support  taat  atratagy  la  conaiatant  with 
tha  development  contractor's  taat  atratagy.  /tg 


QUESTION  DATA  SHEET 


QuMtlon  NuKbar  SPHCTS)  -  001 

QUISTIQW;  Th«  TEMP  ad^quatsly  d««crib«a  .  th«  aoftwar*  t««t  and 
•valuation  procMS* 


ACTIVITY  *  Procur«a«nt  «nd  Operation  Support 

gypyAHATIOlf ;  Tha  aoftwara  taat  and  avaluatlon  proeaaa  Includaa 
objaotivaa,  aaaauraa  of  affaetivanaaa*  organization  raaponaibllitiaa 
and  intarfaeaa*  tha  OT&E  and  0T6E  apacifle  taat  Intarfaeaa,  and  tha 
ovarall  achadula  and  funding  laval  of  tha  taat  proeaaa.  Any  uaa  of 
ZV&V  by  proeuraaant  ahould  ba  daaeribad  along  with  tha  organization 
ralationahlpa  and  axpactad  raaulta.  Tha  TEMP  la  a  eonelaa  daaclptlon 
of  tha  coaplata  ayataa  taat  proeaaa.  but  thara  ahould  ba  adaquata 
attantlon  to  tha  aoftwara  taat  proeaaa.  and  approprlata  rafaranca  to 
othar  planning  doeuaanta  for  aora  datallad  Infornatlon. 

GLOSSARY ; 


RESPONSE  ^MSTRUCT^pW?; 

F/i:  Tha  TEMP  doaa  not  axlat  or  doaa  not  addraaa  aoftwara  taat 
and  avaluatlon 


552£22!5S-SAiigNAi|i 


BSSBQHSE.SQQSMl  _ 

•.rrsTrrr  ,  ■  -  4  •*  i  » 

«  •  ammm  •  ^  •mmrnm  *  •  »  « 


Al-93 


QUESTION  DATA  SHEET 


QuMtlon  NuKb«r  SPH(TS)  -  002 

flUEyryQgl  Th«  ao£tw«r«  tmmt.  procbM  for  D?6E  h«s  followod  th« 
guidolinoa  in  tha  TEMP. 


/iyyiVJT^CS) ;  Proeuraaant  and  Oparablon  Support 

^N^TIOM ;  Tha  TEMP  raflacta  top  laval  input  from  tha  DT&E 
organixatlon  and  ahould  ba  followad  If  tha  TEMP  la  to  ba  an 
important*  high  laval  planning  documant.  Charaetariatica  to  ba 
eonaidarad  includa  achadula*  planning  and  utilization  of  raaoureaa* 
darivation  of  high  laval  maaauraa  of  affaetivanaaa  for  tha  apacifiad 
objactivaa  and  aubobjactivaa*  and  tha  eoamunieation  of  taat  raaulta 
with  othar  taat  organlzationa. 


e 


F/l:  Tha  TEMP  doaa  not  axiat  or  doaa  not  addraaa  aoftwara  DT&E 


S£SESaSS.BA!lS!!il.Sl 


“cQf!PtIiTiLY“o:SAGiii”l..2.3.4.5.6  ■  COMPLETELY  AGREE) 


OUfiSTION  DATA  SHEET 


QuMtlon  Nuabar  SPH<TS>  -  003 

QUESTION;  Tha  abftwara  taat  procaaa  for  OT&E  haa  followad  tha 
guldalinas  in  tha  TEKP« 


ftyTIVITY<S> ;  Procuranant  and  Oparatlon  Support 

EXPLANATION;  Tha  TEMP  raflacta  top  laval  Input  froa  tha  0T6E 
organization  and  ahould  ba  follo%>ad  if  tha  TEMP  ia  to  ba  an 
iaportant,  high  laval  planning  docuaant.  Charactariatiea  to  ba 
conaidarad  includa  achadula#  planning  and  utilization  of  raaourcaa* 
darivation  of  high  laval  aaaauraa  of  affactivanaaa  for  tha  apacifiad 
ob^activaa  and  aubobjaetivaa*  and  tha  coaaunication  of  taat  raaulta 
with  othar  taat  organizationa. 


StSSSARYi 


N 

i 

I 


» 

I 


RESPgNSE_JNST5UCTJONS2 

Tha  TEMP  doaa  not  axiat  or  doaa  not  addraaa  aoftwara  OT&E 


RESPON?^  RAT^QNAIyg; 


3E3PQWSS  SCCKEl _ 

CC0MPLZTiLY"5:SAGRii“”l.2,3,4.3.S  «  COMPLETELY  AGREE) 


I 


Al-99 


QUESTION  DATA  SHEET 


# 

QuMtion  Niwbsr  SPHCTS)  ■*  004 

QUESTION ;  Th«  liiplM*nt«tion  of  tho  softworo  proeooo  by  DT6E  «nd 

OT&E  orgonlzotlona  h«a  b««n  adoquato. 


ACTIVITY^?),;  Procuroaant  and  Oparatlon  Support 

Tha  TEMP  guidallnaa  for  tha  aoftwvra  taat  procaaa  ahould 
raflact  tha  DTAfi  and  0T6E  approachaa.  Tha  actual  iaplaaantation  of 
thoaa  guldallnaa  ia  apaciflad  in  tha  DT6E  and  OTAE  organizational 
plana.  Chaek  tha  affactivanaaa  of  tha  taata  to  atraaa  tha  aoftwara 
coaponanta  in  a  thorough  aannar.  Tha  aaauranea  tha  taata  provida  that 
tha  aoftwara  raquirananta  hava  baan  varifiad  at  a  low  laval  of  datail 
and  validated  in  an  oparationally  rapraaantativa  anvironaant  ia  ona 
aaaaura  of  how  nature  tha  aoftwara  ia  likaly  to  ba  during  early  poat 
daploynant  aupport. 


1.^ 


F/l:  Tha  0T6E  and  OT&E  organizationa  do  not  hava  any  apacific 
aoftwara  taat  plana 


RESPONSE  RATIONALE: 


5555Sli§I-5COREi 


■ :i3AGrtSS  »  1.2,-3..4.3.5  »  COMPLETELY  AGREE: 


& 


QUESTION  DATA  SHEET 

QuMtlon  Nu«b«r  SPM.(TS>  -  005 

QUESTION^-  Th«  tMt  organlzatlona  hav«  lneorpor«t«d  •  «tr«t«gy  in 
th«ir  zoftwar*  tMb  proczMM  for  eoordino'tion  and  aharlng  of  taat 
plana*  proeaduraa*  and  raaulta. 

Asimnssii  All 

JIOH :  Tha  taat  and  avaluatlon  diraetlvaa  raquira  participation 
of  tha  varioua  taat  organizationa  (a;g**  DT&E*  0T6S*  ZVAV)  aeroaa  tha 
coaplata  ayataa  lifa  eyela*  In  addition*  tha  raquiraaant  ia  that 
thaaa  organizationa  coordinata  thair  aetivitiaa  ao  aa  to  ha  aora 
affaetiva  and  thorough.  Thua*  OTfcB  raaulta  ahould  faad  tha  0T6E 
procaaa;  tha  raquiraaanta  of  tha  OT&E  proeaaa  ahould  affact  tha  DT6E 
proeaaa;  tha  ZV6V  proeaaa  and  raaulta  ahould  not .  duplieata*  but 
eoaplaaant  and  aupplaaant  tha  OTAE*  0T6E*  and  davalopnant  contractor 
taating  proeaaa.  Tha  davalopnant  contractor  taating  proeaaa  ahould  ba 
an  intagral  part  of  tha  DTAE  procaaa  and  nonitorad  eloaaly  by  tha 
OTAE  aganey  and  oparation  aupport  activity. 

9V0S?ARY; 


Rg$P0N3E .  IN$TRUCT?gN3 : 


RggPQNSg  RATIONALE ! 


RESPONSE^ SCORE;  _ 

a  •  '*  '5  i  3  j  a  -GM15T  -r  iGacr-. 


Al-97 


QUESTION  DATA  SHEET 


m 

QuMtlon  Number  SPM(TS>  ■*  006 

9VfSTtp>f;  Thm  rmquiremmntm  for  the  development  eontrector  software 
teat  etrategy  are  clearly  specif  lad  in  the  RFP»  SOW,  and/or  CDRLa. 


ACTIVITYiSll  Procurement  and  Operation  Support 

The  best  way  to  achieve  a  good  contractor  teat  strategy 
is  to  require  one.  The  best  way  to.  do  this  is  through  the  RFP,  SOW, 
and  the  CDRLa.  The  form,  methods,  techniques,  schedule,  deliverables, 
transition,  and  so  forth  for  the  development  eontrector  teat  strategy 
should  be  specified  as  part  of  the  eontrectual  requirements. 


(fVOSSARY; 


55SE21!2I-I2SISSSIISI!2i 

F/l:  No  requirement  for  a  development  contractor  software  test 
strategy  is  contractually  specified,  or  what  is  specified 
is  totally  inadequate. 


RESPONSE  rat;on^4^; 


•yggsrvqj  2CC3S: 

'Tc5MPLiTiLY“0ISAGRii”«"l.2.3,4,5.S  «  COMPLETELY  AGREE) 


Al-98 


QUESTION  DATA  SHEET 


QuMtlon  Number  SPMCTS)  -  007 

flUSSIISlil  The  uee  of  en  orgenixetion  for  aoftwere  test  IV&V  aupport 
haa  been  effective. 


|^CT^V^TY<?) :  Procurement  and  Operation  Support  * 

;  The  IV&V  function  ia  not  alwaya  required  or  appropriate. 

For  aiaaion  critical  ayatema  with  aignificant  aoftware  it  ia  uaually 
required.  The  aoftwere  XVAV  mey  be  aeperate  from  or  a  part  of  a 
ayatem  IV&V  effort.  For  ZV&V  to  be  effective  it  muat  uaually  be 
applied  early  in  the  aoftware  life  cycle  and  compriae  a  aignificant  ' 
Ce.g.»  10  TO  30  percent)  portion  of  the  aoftware  development  coat. 
There  are  apecific  inatancea  where  IVfcV  can  be  uaed  to  aolely  aupport 
DT&E  and/or  OT&E  in  their  apecific  functiona.  in  which  caae  the  IV6V 
function  ia  leaa  comprehenaive  and  coatly.  Thia  can  be  en  eapecially 
effective  way  to  obtain  detailed  aoftware  atreaa  teating  and 
evaluation  which  might  not  be  done  becauae  of  a  ahortaga  of  teat  ' 

organixation  aupport  peraonnel. 


R^SPONS^  TN^TRUgTIQNS: 

F/l:  No  ZV&V  function  haa  been  defined,  but  the  ayatem  ia  a 

miaaion  critical  ayatem  with  aignificant  aoftware  functiona. 


RESPONSE  .RAT^gNALf : 


— - _ 

'7c5MPLi7iLY”0I3AGRii”«”l.2.3.4.5.6  ■  COMPLETELY  AGREE) 


Al-99 


*  »>  <■**«  *S 


QUESTION  DATA  SHEET 


QuMtlon 


VMBb«r  SPM(TS>  -  00 


aySSIISNl  Th«  ovarall  planning  for  •oftwar*  teatlng  haa  baan 
adaquata. 


All 

Bff ;  Tha  eoablnation  of  proeuraaant#  dovwlopaant  contractor, 
and  operation  aupport  aoftwara  taat  planning  ahould  raflact  an 
integrated  atrategy  for  accoapliahing  the  eoft%Mure  teat  procaaa 
throughout  tha  aoftware'a  complete  life  cycle.  The  varioua 
activitiea'  teat  plana  ahould  identify  and  empheeixe  thoae  aapacta 
of  primary  raaponaibility.  In  particular,  auch  plana  ahould  identify 
teat  itama,  featurea  to  be  teated,  faaturea  not  to  be  taatad, 
relationahip  of  the  teat  itama  to  the  baaeline  being  teated  and  tha 
functional  ayatam  requiramenta,  taat  approach  and  mathodologiaa 
employed,  paaa/fail  criteria,  apecific  taat  taaka  (WBS),  taat 
environment,  taat  raaponaibility  and  raaourca  requireaenta,  and  taat 
eehedulaa.  The  taat  plan,  teat  deaeriptiona,  teat  log,  teat  raaulta, 
incidance  reporta,  and  any  other  teat  documentation  to  be  produced 
ahould  be  deacribed  in  the  aoftwara  teat  plana. 


RggPqWSE  ^NSTHUCTiqWS: 


RESPONSE  RATIONALE; 


555EQli2S_SSQ55i 


AGREE  » 


7C?!PLE”"S*  V 


agree: 


Al-lOO 


QUESTION  DATA  SHEET 


QuMtlon  Nu»b«r  SPH<T8>  -  009 

QUESTION  ^  Th«  softwar*  tMt  approach  and  aathodologlaa  aaployad  ara 
elaarly  daacribad  In  tha  aoftwara  taat  docaantation  and  appaar  to  ba 
af f actl va • 

tsimnssii  All 

6jffit.ANATI0N:  Tha  approach  ahould  ba  daacribad  for  aach  major  group  of 
faatiiraa  to  ba  taatad.  Tha  major  actlvltiaa*  tachniquaa*  and  toola 
which  ara  to  ba  uaad  ahould  ba  daacribad  In  anough  datall  to  Idantlfy 
major  taatlng  taaka  and  aatlmatlon  of  tha  tlma  ragulrad  .to  do  aach 
ona.  Tha  minimum  dagraa  of  complatanaaa  ahould  ba  apaclflad  along 
with  tha  tachnlquaa  <a.g.«  path  covaraga#  atatamant  covaraga*  domain 
apaca  covaraga)  uaad  to  judga  complatanaaa.  Aceaptanca  eritarla 
(a.g.,  fault  fraquancy)  ahould  ba  apaclflad  aa  wall  aa  taatlng 
conatralnta  aueh  aa  taat-ltam  availability*  taat  raaourca 
availability*  aehadula  daadllnaa*  and  funding  lavala.  Taatlng 
tachnlquaa  Includa  coda  ravlawa*.  atructura  analyala*  atatle  program 
quality  analyala*  dynamic  path  analyala*  covaraga  analyala*  aaaartlon 
chacklng*  aymbolle  dabugglng*  mutation  taatlng*  and  ragraaaion 
taatlng.  Such  tachnlquaa  can  ba  appllad  with  varying  auccaaa  aeroaa 
tha  taat  phaaaa  of  daalgn  varlflcatlon*  unit  and  moduia  taat*  CSC! 
and  ayatam  Intagratlon  taat*  PQT/FQT*  ayatam  taat*  and  mlaalon  taat. 
Application  of  taatlng  tachnlquaa  aeroaa  taat  phaaaa  and  daaeriptlona 
of  how  tha  overall  taat  objactlvaa  ara  to  ba  aehlavad  ahould  ba 
elaarly  apaclflad  in  tha  taat  documantatlon. 

GtgSSARY^ 


3x§22i!§s_IN§IS2SI5222i 


RESPONSE  rationale: 


5KBQ3§i-.2COR5j. _ _ 

“rCOMPLiTSLY'oiSAGREE  ■  1.2. 3. 4. 5. 6  •  COMPLETELY  AGREE) 


QUESTION  DATA  SHEET 


QuMtion  Nusb^r  SPH(TS)  -  010 

OmSTIQN;  Th«  software  faaturaa  to  bo  tootod  ond  not  to  bo  tostod  «ro 
cioorly''dooeribod  in  tho  oo£tworo  toot  doeuaontotlon. 


ASIIVimSll  All 

ATI9H '  ooftworo  toot  doeuaontotlon  ohould  Idontlfy  footuroo 
to  bo  tootod*  footuroo  not  to  bo  tootod*  ond  oil  eoablnotlono  of  ouch 

footxuroo  ocrooo  tho  toot  eoooo. 


I 

i  SMSSfiS2i 


e 


5g5£Sl!25-Il!SI51iSlI21!2i 

F/l:  Softworo  toot  doeuaontotlon  dooo  not  Identify  footuroo  to  bo 
tootod  ond  not  tootod. 


SSSBSSSS-SATIQNAtgi 


5§=£SL’S2_3cc3S2 

*7c5MPuiTiLY“D'iSAGiii"»”l,2,3,4.5.6  ■  COMPLETELY  AGREE) 


Al-102 


BGsnzmsnnR 


.1- 


QUESTION  DATA  SHEET 


QuMtlon  Number  SPH(TS}  -  Oil 

9yf ST^QN The  treceeblllty  of  eoftwere  features  tested/not  tested  to 
the  software  functional  requlreaents  la  described  in  the  software 
test  docuaentstion. 

All 

software  test  documentation  should  provide  assurance 
of  which  functional  requirements  of  the  software  have/have  not  bean 
satisfied.  A  clear  association  of  the  functional  requirements  with 
the  software  features  and  the  related  teats  is  a  masor  step  toward 
providing  that  assurance.  Typically*  a  matrix  or  written  description 
is  provided.  Use  of  a  cross  reference  among  data  dictionaries  for 
requirements*  faaturaa*  and  testa  is  anothar  way  such  information  can 
be  presented. 


glossary; 


REgPgMSS^INSTRUCTIQMSi 

F/i:  No  croaa  reference  exists  among  software  features, 
teat  cases*  and  functional  requirements 


REgPONSg^RATigNAtgi 


RESPONSE  SCOREi  _ 

”“c5NPLiTiLY”5lSAGRSS  ■  1.2.3.4.5.S  «  COMPLETELY  AGREE) 


Al>103 


jnpflUfMuCTunununc 


QUESTION  DATA  SHEET 


if 


QuMtlon  Nu»b«r  SPHCTS)  '**  012 

QUESTION ;  Thtt  aoftwcrtt  tttst  dAlivarablaa  mrm  adaquataly  ap«clfi«d  In 
tha  aoftwara  taat  docuaantatlon. 

bSlDLlllLllL  All 

aoftwara  taat  doctiaantatlon  ahould  provlda  a 
eoaplata  Hat  and  daaeriptlon  o£  all  aoftwara  taat  dallvarablaa.  A 
aoftwara  taat  plan  la  tha  aoat  logical  location  for  auch  Information. 
It  ahould  alao  ba  claarly  atatad  how  tha  dallvarablaa  ralata  to  tha 
CDRLa/DIDa  In  tha  caaa  of  tha  davalopmant  contractor. 


G^OSJARYi 


3=5S2N2I-Il!5I32SII2N2i 


RESPONSE  RATIONAL^; 


RESPONSS^SCOREi 

“<C0KPLifELY”DlSAGRii”«“l,2,3,4,S,S  ■  COMPLETELY  AGREE) 


QUESTION  DATA  SHEET 


QuMtlon  NuBbcr  8PH(TS>  -  013 

OyESTipN!  softwBra  taat  erltarla  usad  to  dataralna  whathar  aach 
taat  ha«~paaaad  or  fallad  ia  claarly  apaciflad  in  tha  aoftwara  taat 
doeunantation . 

tsii^inssii  All 

aoftwara  taat  doeuaantation  ahould  provlda  paaa/fall 
crltaria  for  aach  taat  and  aach  aoftwara  faatura  to  ba  taatad.  Tha 
erltarla  ahould  ba  aa  ob^activa  aa  poaalbla.  Exaaplaa  of  critaria 
Includa  pareantaga  of  atataaanta  taatad »  parcantaga  of  logic  pa  tha 
taatad*  and  faulta  (fraquancy*  nuabar  of  critical/non-critical) 
allowad. 


glossary: 


SSS£S!iSS.INSIBySIIS!fSl 


F/l: 

50» 

or 

aora 

of 

tha 

taata/faaturaa 

hava 

inadaquata 

critaria 

S/2: 

40» 

to 

SOX 

of 

tha 

taata/faaturaa 

hava 

inadaquata 

critaria 

D/3: 

30H 

to 

40X 

of 

tha 

taata/faaturaa 

hava 

inadaquata 

critaria 

C/4: 

20» 

to 

30X 

of 

tha 

taata/faaturaa 

hava 

inadaquata 

critaria 

3/5: 

109C 

to 

20X 

of 

tha 

taata/faaturaa 

hava 

inadaquata 

critaria 

A/6: 

OX 

to 

lOX 

of 

tha 

taata/faaturaa 

hava 

inadaquata 

critaria 

R§SPgNSg_RATigNALSi 


vCOMPLITSLY  DISAGREE  »  1, 2*3, -i. 5. 5  »  CCMPL2TEL7  AGREE) 


QUESTION  DATA  SHEET 


QuMtlon  Nuaib«r  SPH<TS>  *  014 

92ESIISN1  Th«  p«rsonn«l  group*  rMponslblo  for  tho  softwaro  boats  aro 
adaquataly  Idantlflad  In  tha  aoftwara  taat  doeuaontation . 

ESnviIXiSli  All 

aoftwara  taat  doeuaantatlon  should  claarly  Idantify 
paraonnal  groups  who  ara  raaponaiblo  for  tha  various  aoftwara  taata 
acroaa  all  tha  taat  phaaaa.  Zn  aoaa  eaaaa  tha  individual  prograaaar 
will  ba  raaponaibla*  in  othar  eaaaa  a  eoaplata  indapandant  taat  group 
Bay  ba  raaponaibla.  Raaponaiblltiaa  ineluda  aanaging*  daaigning* 
praparing#  axaeuting*  aonitoring,  chocking,  raaolving  anonalias,  and 
aecaptanea/approval .  A  aoftwara  taat  plan  is  tha  most  likaly  aourca 
of  this  inforaation,  although  individual  taat  eaaa  dascriptiona  ara 
also  a  poaaibla  aourca. 

sirfiss^sn 


5SS£Sjf9S.Il!SISSSII9NSi 


F/i: 

SOX 

or 

Bora 

taata 

hava 

E/2: 

40X 

to 

SOX 

taata 

hava 

D/3: 

30X 

to 

40X 

taata 

hava 

c/4: 

20X 

to 

30X 

taata 

hava 

B/5: 

lOX 

to 

20X 

taata 

hava 

A/6: 

OX 

to 

lOX 

taata 

hava 

B£9S91!SI.£mSN^Sc£i 


RESPONSE  SCORE: _ 

”  <  COfiPLETEL  y”  5  IS  ACRii"  ■  ”  1 . 2 , 3 , 4 


no  raaponaiblo  group  idantlflad 
no  raaponaibla  group  Idantifiad 
no  raaponaibla  group  idantifiad 
no  raaponaibla  group  idantifiad 
no  raaponaibla  group  Idantifiad 
no  raaponaibla -group  idantifiad 


I 


9.6  -  COMPLETELY  AGREE) 


QUESTION  DATA  SHEET 


QuMtion  Nuxbar  SPH(TS>  ■*  015 

QUESTION  i  Th«  high  risk  «sau»ptlona  of  th«  ooftwor*  tasting  approach 
along  with  contingancy  plana  for  aach  such  assumption  is  adaquataly 
daacrihad  in  tha  aoftwara  taat  documantation. 

ASIiyilXi§2i  All 

E^glfAy^XIpy,;  Tha  aoftwara  taat  documantation  should  claarly  idantify 
any  araaa  of  high  risk.  Examplaa  of  high  risk  taata  includa  thoaa 
eaaaa  which  would  raquira  intaragancy  Ca.g.#  intaraarvica)  allocation 
of  raaourcaa  such  aa  taat  rangaa*  aquipmant^  or  paraonnal  for  which 
tha  availability  is  scares.  A  achadula  daisy  in  such  a  taat  might 
causa  a  larga  rippla  affact*  not  only  in  tha  aubjact  ayatam  taat 
achadula*  but  in  tha  taat  achadula  of  ayatama  compating  for  tha  asms 
taat  raaourcaa.  Altarnata  plana  for  handling  auch  difficultiaa 
(raaourca  availability*  funding  laval*  tachnical  problama)  should  ba 
daacribad.  A  aoftwara  taat  plan  is  tha  aoat  likaly  aourca  for 
idantifying  tha  high  risk  taat  eaaaa  and  high  laval  contingancy 
plana.  Tha  Individual  taat  caaa  daacription  would  probably  provida 
mora  datail  aa  to  why  tha  taat  caaa  la  high  riak. 

GLOSSARY: 


RESPONSE . instructions: 


F/i: 

SON 

or 

mora 

of 

tha 

high 

riak 

taata 

hava 

no 

contingency* 

plan 

or 

high 

riak  taata  axiat 

but  ara  not  idantifiad 

E/2: 

40m 

to 

SON 

of 

tha 

high 

riak 

taata 

hava 

no 

contingency 

plan 

0/3: 

30m 

to 

40k 

of 

tha 

high 

risk 

taata 

hava 

no 

contingancy 

plan 

c/4: 

20m 

to 

30k 

of 

tha 

high 

risk 

taata 

hava 

no 

contingency 

plan 

B/S: 

lom 

to 

20k 

of 

tha 

high 

risk 

taata 

hava 

no 

contingancy 

plan 

A/6: 

Ok 

to 

10k 

of 

tha 

high 

riak 

taata 

hava 

no 

contingency 

plan 

555ES*l§S-5AIIQNAtgi 


RESPONSE  SCORE:  _ 

”7cOMPLSTSLY“5iS AGREE  ■  1.2. 3. 4. 5. 6  ■  COMPLETELY  AGREED 


Al-107 


QUESTION  DATA  SHEET 


QuMtlon  Muab«r  SPM(TS>  -  016 

QUlSTIOKl  Th«  achadula  for  ooftworo  toot  allMtbnM  1«  adoquatoly 
spoclfiod  in  tho  aoftwaro  to«t  docuaontation. 


tSIDLniLllL  All 

:  Th«  aeftwaro  t««t  docuMntation  should  eloarly  identify 
tho  uilootonM  for  oach  t««t.  Thoao  niloatonos  mxm  noraally  part  of 
tha  software  taat  pi  an /approach*  tha  Software  Project  Schedule  which 
ia  a  aubpart  of  tha  ayataa  WBS  aehadula*  or  tha  ayataa  davalopaant 
achadula.  Tha  taat  ailaatonaa  and  thair  relationship  to  tha  overall 
software  and  ayataa  davalopaant  schedule  should  be  specified. 


SkQSS&RYl 

Xff j^_g4^aatQnaa.  Coaplation  of  design*  preparation*  execution, 
resolution  of  anoaaliaa*  doc\iaantation*  accaptanca  are  typical 
ailaatonaa. 


V 


Rg^ppNS^ .iw?TRUCT;gws; 


F/i: 

sox 

or 

sore 

of 

tha 

software 

taste 

have 

no 

aila'etona 

achadula 

E/2: 

40X 

to 

SOX 

of 

tha 

software 

testa 

have 

no 

ailaetona 

achadula 

D/3: 

30X 

to 

40X 

of 

tha 

software 

taste 

have 

no 

ailaetona 

achadula 

C/4: 

20X 

to 

30X 

of 

tha 

software 

teats 

have 

no 

milastona 

achadula 

B/S: 

lOX 

to 

20X 

of 

tha 

software 

taste 

have 

no 

milestone 

achadula 

A/6: 

OX 

to 

lOX 

of 

tha 

software 

taste 

have 

no 

milaetona 

achadula 

RgpPQNSE. gATiqWALf : 


5§§S91!§l«scgRE2 

”<C0MPLETELY”DISAGRii"“l,2*3,4,5,6  •  COMPLETELY  AGREE) 


Al-106 


QUESTION  DATA  SHEET 


QuMtlon  Nuxbar  SPH(T^>  -  017 

QUESTION;  Softwara  tasting  la  adaquataly  prlorltlzad  In  tha  aoftwara 
taat  approach  according  to  alaaion  criticality  concarna. 


ACTIVITYCS);  All 

SjCfLANATIOgj  Tha  aoftwara  taat  approach  ahould  prlorltiza  tha 
aoftwara  taatlng  proeaaa  according  to  alaaion  critical  faaturaa.  For 
axaapla^  If  cartaln  aoftwara  faaturaa  ara  critical  to  tha  alaaion 
rallabllty  of  tha  ayataa  or  parhapa  tha  aafaty  of  tha  ayataa 
paraonnalf  than  thoaa  faaturaa  ahould  racaiva  a  hlghar  priority. 
Hlghar  priority  faaturaa  may  raqulra  aora  rlgoroua  taata,  norm 
objactlva  aaaauraa  of  taat  aaauranca*  longar  taat  achadula*  and  a 
mora  vialbla  taat  raportlng  proeaaa. 


gyOSSARY: 

A-£ff .  Any  faatura  of  tha  ayataa  which  will 
pravant  tha  coaplatlon  of  tha  alaaion  objactlva  or  lapacta  tha  aafaty 
of  tha  tha  paraonaal  who  ara  part  of  tha  alaaion  If  It  la  not 
davalopad  eorractly  . 


5i5£SB§l-I?!§ISy£I5Q!2§i 

f7I:  No  prioritization  of  taata  la  apparant  froa  tha  aoftwara 
docuaantatlon.  or  tha  daflnad  prioritization  la  not  balng 
followad 


RESPONSE  RATIONALE! 


5S§£QfiSE_SC0REi  _ 

“  .':cH?TliTiLY”5l3AGRSH  »  1.2, 3. 4.5. 3 


2CI1PL2.T2LV  .iGJ 


Al-109 


QUESTION  DATA  SHEET 


I 


QuMtlon  Nuabar  SPM<TS)  -  Old 

QUESTION ;  Th«  softwara  taat  anvironaant  la  adaquataly  Idantiflad  In 
tha  aoftwara  taat  doeuaantatlon  and  la  adaquata  for  accoapllahlng  tha 
raqulrad  taatlng. 

(jQT^yiTYCS);  All 

gXPy^NATlON :  Tha  aoftwara  taat  doeuaantatlon  ahould  Idantlfy  tha 
aoftwara  taat  anvironaant  Including  hoat  aaulatlon/alaulatlon» 
aoftwara  banch  taatlng  aqulpaant*  Intagratad  laboratory  anvlronaanta* 
and  oparatlonal  alaalon  taat  anvlronaanta . 


GLOSSARY; 


•2H£QNSg.IN5I5UCTigNSi 

F/lt  No  Identification  of  tha  aoftwara  taat  anvironaant  la 
docuaantad»  or  tha  anvironaant  la  totally  Inadaquata 


RggPQN^g  rational; ; 


ggSPQ^fSE.SCOgE; _ 

"“cOKPLlTiLY”  IS  AGREE  «  1,2. 3. 4. 5. 5 


30NPL27ELV  AGREE) 
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QUESTION  DATA  SHEET 


QuMtion  Number  SPH(TS)  *  019 

QUESTION;  Th«  configuration  nanaganont  of  tha  aoftwara  taat  procaaa 
la  adaquata. 


activity <S):  AH 

gXPt^AATION;  All  actlvltiaa  hava  aoaa  raaponalbility  for  aoftwara 
taat  and  avaluation.  Aa  aueh#  aaeh  activity  haa  a  raaponalbility  for 
ita  particular  anphaaia  to  maintain  approprlata  configuration 
aanagaaant  of  tha  taat  procaaa  and  ita  docuaantation .  In  particular, 
tha  procuraaant  0T6E  and  0T6E  taat  plana,  approachaa,  daacriptiona, 
and  raaulta  ahould  ba  undar  tight  configuration  aanagaaant.  Likawiaa, 
tha  davalopaant  contractor 'a  aoftwara  taat  docuaantation  ahould  ba  a 
contract  dallvarabla,  parhapa  aa  a  CSCI  dapanding  upon  tha 
criticality  of  tha  aoftwara.  Tha  oparation  aupport  aoftwara  taat 
docuaantation  ahould  ba  carafully  controllad  throughout  tha  poat 
daployaant  aupport  of  tha  aoftwara. 

gLOgSARY; 


e 


gESPgNSE_INSTRUCTigNS2 

F/i:  No  configuration  aanagaaant  of  tha  aoftwara  taat 
procaaa  and  ita  docuaantation  haa  baan  plannad 


REgPQNgE^RATigNA^g^ 


Al-111 


QUESTION  DATA  SHEET 


I 


QuMtlon  Euabttr  SPMCTS)  -  020 


QUESTION;  Th«  transition  o£  ths  softwsrs  tsst  strstssy  from  ths 
dsvslopmant  contractor  to  tha  operation  support  activity  has  baan 
adequately  addraaaad  in  tha  software  test  docuaantation  and  tha 
proeuraaant  software  test  plans. 


&simni9>i  All 


gX?LANATION ;  Tha  transition  of  ss  such  of  tha  softiMra  test  strategy 
as  possible  must  be  planned  from  tha  contract  raqulrasants  through 
tha  program  aanagasant  responsibility  transfer.  From  tha  proeuraaant 
PKP,  TEHP«  CRLCHP  <CRISP,  0/S  CMP),  RFP,  and  CORLa  through 
davalopaant  contractor  and  operation  support  activity  software  test 
planning «  tha  transition  must  be  integrated  as  s  critical  part  of 
tha  software  dalivarabla  for  supportability  purpoi 


GLOSSARY! 


0k 


F/i;  No  transition  of  tha  davalopaant  contractor  software  test 

plan,  teat  caaaa*  test  approach,  or  test  tools  ia  docuaantad 


RESPONSE  RATIONALE! 


5i5BS!f5S-5S2BIi _ 

"<COMPLSTiLY'’0ISAGRis  »  1.2. 3. 4. 5. 5  ■  COMPLETELY  AGREE) 
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SOFTWARE  PROJECT  HANAG^'^EHT  PROJECT  INTERFACES 


Th*  quMtlona  SPH(PI)-001  through  SPH<PZ}*016  «ddr«s«  adoquacy 
o£  coftwaro  pro 3 vet  nanogamont  axtarnal  intarfacaa  among  tho 
procuroaantf  davolopaant  contractor,  operation  support  ,  and 
intaraarviea  organization  alaaanta  aa  ia  appropriate.  These  project 
aenegeaent  interfaces  ere  eatebliahad  ao  that  project  information  can 
be  acre  efficiently  coaaunicatad.  These  interfaces  include  the 
physical  relationship  aaong  the  organization  eleaenta,  and  the  high 
level  logical  relationship  of  the.  organization  eleaenta  to  the 
project's  functional  requireaenta. 

There  are  generally  two  types  of  interfaces  which  are  important 
to  evaluate  for  any  project:  internal  organization  interfaces,  and 
organization  interfaces  across  external  boundaries.  The  project 
interface  evaluation  prlaarily  focuaea  on  how  well  the  axtarnal 
interfaces  and  basic  project  organization  to  support  those  interfaces 
for  each  major  organization  component  facilitate  production  of  a  high 
quality  software  product. 

Character iatica  which  should  be  evaluated  are  the  number  of 
external  intarfacea,  size  of  interface  organizations,  various  working 
groups'  interfaces,  and  application  of  directives  and  regulations  to 
control  the  coordination  among  intarfacing  organizations.  Typical 
interface  working  groups  include  the  Computer  Reaoufeea  Working  Group 
(CRW6>,  the  Teat  Planning  Working  Group  (TPWG>,  end  the  'Interface 
Control  Working  Group  (ICWG). 


Al-113 


QUESTION  DATA  SHEET 


QiMStion  Number  SPH(PX> 


9UgS7|0N ;  The  eyatem  program  office  extarnal  interfeeea  ere  adaquate. 


ACT|VyTY<3) :  Procurement 

gyg^AgATIOW ;  The  program  office  Interfacea  with  the  impleaenting 
commend »  the  uaing  commend «  the  aupporting  command#  the  training 
commend#  the  teat  and  evaluation  ageneiaa  for  DT&E  and  0T6E#  apecial 
government  contract  eervice  ageneiaa  aueh  aa  The  MXTNE  Corporation# 
and  the  development  contractor.  Theae  external  interfacea  generally 
ere  concerned  with  Air  Force  policy#  adherence  to  regulationa  and 
direetivee#  interaerviee  implieatlona  of  the  eyatam  under 
development#  general  computer  raaource  management  iaauea*  and 
reaource  (peraonnal#  aystema#  time#  funds)  planning  and  uaa.  The 
program  manager  ia  the  primary  head  of  the  program  office  and  muat 
Inaure  that  the  program  office  worka  with  the  varioiia  commanda  and 
ageneiaa  to  eatabliah  the  meana  to  implement  the  ayatem  acquiaition 
dictated  in  the  PHD  and  deacribed  in  the  PHP.  Theae  interfacaa  help 
to  define  the  technology  eonatrainta  on  the  ayatem  and  ita  aoftwara. 
including  what  advanced  computer  technology  will  be  required  to  be 
applied.  The  initial  emphaaia  on  aoftwara  aupportability  would  be  in 
the  PMP. 

GLOgSAgYi 

An  Air  Force  procuring  activity#  headed  by  s 
program  manager#  end  eatabliahed  within  a  product  diviaion  (a.g.. 
Aeronautical  Syatema  Diviaion)  early  in  the  damonatration  and 
validation  pheae  for  the  purpoae  of  fulfilling  the  program  management 
reaponaibilitiea  deacribed  in  the  ayatem  PMO. 


SggpCNSS  INgTaUCTjgN? : 
gSgPQN:gf  I^ATJQNALE: 


P5SPQNSS  SCORE; 

”<c5MPLiTiLY”DlSAGRii'’»”l,2,3#4#5,6  ■  COMPLETELY  AGREE) 
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QUESTION  DATA  SHEET 


OuMtion  Nu«b«r  SPH(PI)  -  002 

QUEST  I QN;.  Th«  luplamanting  co»M«nd  •xtarnal  lnt«r£«eM  ar*  adaquata. 


ACTIVITY <3)  ;  Procuraaant 

gXPLANATION ;  Tha  laplaaanting  coaaand  worka  with  tha  prograa  offica 
to  aaka  aura  tha  PKP  addraaaaa  propar  davalopaant  laauaa»  and  with 
tha  aupporting  and  ualng  conaanda  to  aaaura  that  tha  oparatlonal  and 
support  iaauaa  ara  proparly  addraasad.  If  thara  ia  a  davalopaant  ZV6V 
function#  than  tha  iaplaaanting  coaaand  will  Intarfaca  with  tha  IVfcV 
contractor /agancy  to  aaaura  propar  coordination  aaong  tha  program 
offica  and  tha  davalopaant  contractor  activity.  Tha  iaplaaanting 
coaaand  ia  tha  priaary  contract  aonitor  and  technical  raviawar  of  tha 
davalopaant  contractor  activity  taaka.  Tha  laplaaanting  coaaand 
pro^act  prograa  aanagar  auat  halp  to  dafina  tha  technology 
conatrainta  on  tha  ayataa  and  its  aoftwara#  including  what  advanced 
coaputar  technology  will  ba  required  to  be  applied.  Tha  iaplaaanting 
coaaand  also  participataa  in  working  groupa  aueh  aa  tha  CRWG.  ICW6# 
and  TPWG.  Tha  initial  aaphaaia  on  software  aupportability  would  ba  in 
tha  PHP. 

lEBlggfgjr^DS.SgggfSdj  coaaand  raaponaibla  for  aequiaition  of 
a  ayataa:  davalopaant#  operation#  and  aaintananca  till  tranafar  of 
prograa  aanagaaant  to  tha  aupporting  coaaand  at  PHRT  and  turnover  of 
tha  ayataa  operation  to  tha  uaing  coaaand  soaatiaa  prior  to  PHRT.  Tha 
iaplaaanting  coaaand  ia  usually  AFSC  or  a  HAJCOH. 


5§§£9i!SI-5J!§IS!iSII21!5i 

REgPQW^E  .RATIOWALg; 


S§§E935I_3S9BSi  — _ 

~<C0MPLiTiLy~5zSAGREE  -  1#2#3#4#S#6  ■  COMPLETELY  AGREE) 
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mimnrmiinrm  a  1 1  m 


QUESTION  DATA  SHEET 


QuMtlon  Nuab«r 


SPH(PI>  -  003 


QUE3T1;QN;  Th«  using  command  •xtsrnml  Intmrfacmm  mrs  admqumts. 


;  Opsrmtlon  Support 

fXy VAN/^TIOW ;  Ths  using  <opsrsting>  eosamsnd  has  ths  rssponslbility  of 
opsrstlonsl  dsploymsnt  of  s  systsm*  subsystsm*  or  itsa  of  squipmsnt. 
Ths  using  eosmsnd  hss  sxtsrnsl  intsrfacss  with  ths  supporting 
eomxsnd*  implsmsnting  eomssnd*  progrsa  offies*  snd  working  groups. 
During  ths  sequisition  of  ths  systss  to  bs  opsrstsd»  tho  using 
eommsnd's  prixsry  intsrfses  rols  is  ss  monitor  to  ssolst  in  dsriving 
ths  system  rsquirsmsnta  nscssssry  to  msks  ths  systsa  opsrstionally 
sffsetivs.  During  ths  post  dsploymsnt  softwars  support,  ths  using 
eommsnd  Intsrfscsa  with  ths  supporting  command  snd  various  working 
groups  conesrning  aoftwsrs  block  rslsssss. 


Using  Commands  Ths  command  (or  eoaasnds  and  contractor  support) 
rssponsibis  for  ths  opsrstionsl  smployasnt  of  ths  sequirsd  systam 
usually  3ust  prior  to  PHRT. 


mk 


55S£2!!5I-Il!SIS2SII21!2i 


I 

RESPONSE  RATIONALE: 


5SSSSNSi.SSSR£i  _ 


'TCMPLSTELY  DISAGREE  »  1.2.3,4.5.S  »  COMPLETELY  ACRES) 
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QUESTION  DATA  SHEET 


QuMtlon  Nuabttr  SPHCPZ)  -  004 

flUKIIQNl  Th«  aupportlng  command  mxtmrnml  Intmrfmcmm  arm  adaquata. 


;  Oparation  Support 

f3ff ly^yATIQM ;  Tha  supporting  command  axtarnsl  intarfaeaa  ineluda:  tha 
program  offiea*  Implamanting  command*  uaing  command*  i#orking  groups 
(CPW6*  ZCWG*  TPVG)  as  appropriata.  Tha  primary  funetiona  of  tha 
intarfaeaa  ara  to  aaaura  that  tha  ayatam  as  dalivarad  ia  aupportabla 
and  that  appropriata  support  anvironmant  ia  acquirad  for  uaa  during 
poat  daploymant  support. 


GkSSS&SY: 

gyppoyt^^P?  ■  I  command  (or  commanda  and  contractor 

support)  raaponaibla  for  tha  conf igtiration  maaagaaant*  logistics 
support*  and  othar  kinds  of  diract  support  raquirad  by  a  ayatam 
during  oparational  uaa.  9^fieial  raaponaibility  ia  aaaumad  from  tha 
davalopmant  contractor  activity  at  PMRT. 


5fiffi21!25-IJ!SI52SI221i2i 


5fS£S2SI.52IIfil!2Ui 


RSfPONSg  SCORER  _ 

"'CCJ»PLSTlLy“’?ISAGSSE  »  1.2. 3. 4. 5, 6  «  COMPLETELY  ACRES) 
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QUESTION  DATA  SHEET 


QuMtion  Nu»b«r  SPH(PI)  *  OOS 

SUESIISNi  Th«  training  eoaaand  axtarnal  intarfacaa  ara  adaquata. 


(3) ;  Procuraaant  and  Oparatlon  Support 

training  eoaaand  raviawa  ayataa  doeuaanta  and 
initiataa  training  aupport  planning  and  avaluation  aa  approprlata, 
and  providaa  and  adainiatara  training  prograaa  to  aupport  ayataaa. 
Tha  intarfaeaa  for  aueh  training  could  includa:  iaplananting,  uaing« 
aupport ing  coaaanda;  tha  DTAE  and  OTbE  aganeiaa. 


SirSSSASYi 

Xyg ining.Coaaand*  Tha  eoaaand  (a.g.»  HQ-ATC>  raaponaibla  for 
providing  planning*  avaluation  *  conduct*  and  adainiatration  of 
training  prograaa  and  training  raquiraaanta. 


s_ 


RESPONSE , 


SiSSSSSi-SATigNAieli 


RESPONSE  SCORE; 

'<c5M?LSTiLY"5lSAGRii~  1*2*3*4*9*6  -  COMPLETELY  AGREE) 
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QUESTION  DATA  SHEET 


Quest ion  Nusbsr  SPM<PI>  -  006 

QUESTION The  dsvslopssnt  contractor  sxtarnal  Intarfacsa  are 
adaquata . 


<S.> Davalopaant  Contractor 

Tha  davalopa  ant  contractor  primary  axtarnal  Intarfacaa 
ara  with  tha  program  office,  implementing  command,  teat  and 
avaluation  aganciea  (OT&E,  0T6E,  ZV6V>,  supporting  command,  using 
command,  and  working  groups  as  is  appropriate.  Tha  ma^or  function  of 
tha  intarfacaa  are  communication  of  development  raqulremants  and 
status,  conduct  of  integration  and  operational  testa,  and  transfer  of 
tha  developed  ayatem  to  the  operetion  support  activity. 


gLOgSARYi 

prime  contractor  and  any 
aubcontr actors  reaponaible  for  the  full  scale  development  of  the 
aoftware  ayatem. 


RESPONSE  IN3TRUCTT9NS : 


SSSSSi!SI.SAIISNAi,gi 


n 


oe^ooveg  gCORS ! 

;COi'lPL£TEL f  JISAGREE  *  1,2,3,4,S,6  •  COHPLETEL'if  AGREE) 


9 
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QUESTION  DATA  SHEET 

OuMtlon  Nuabar  SPH(PI> 


# 

007 


QUESTION :  Th«  Davalopaant  TMt  ~«nd  Evaluation  (OTAE)  organization 
axtarnal  intarfacaa  ara  adaquata. 


ACTIVITY <S) ;  Proeuraaant 

gyp^^NATION ;  Tha  iaplaaanting  eoaaand  ia  raaponaibla  lor  DT6E 
aanagaaant.  Tha  davalopaant  contractor  and  tha  iaplaaanting  eoaaand 
jointly  conduct  tha  aarly  part  of  OTAE.  Tha  apaeifiea  of  tha  OTAE 
nanapaaant  ia  doeuaantad  in  tha  TEMP.  DTAE  data  auat  ba  provided  to 
tha  OTAE  agency.  Soae  of  tha  funetiona  of  tha  intarfacaa  ara  to 
coaaunieata  evaluation  of  contract  apaeifieationa*  ayataa/aoftwara 
daficianeiaa«  intaroparability  capability#  and  aatiaataa  of  tha 
ayataa'd  operational  reliability,  aupportability,  availability,  and 
aafaty.  Sea  APR  AO- 14  for  nora  information. 

gLOSSABYl 

BXII,..8ggfP^»*tion.  Tha  organization  alaaahta  (primarily 
implaaanting  command  and  davalopaant  contractor)  raaponaibla  for 
damonatrmting  that  tha  ayatam  (including  hardware  and  aoftwara) 
daaiqn  and  davalopaant  ia  coaplata,  that  daaign  rlaka  have  bean 
minimizad,  and  that  tha  ayatam  will  perform  am  required 
apacif  lad .  % 


RSSE9KSS.Il!SI&S{SIIQ!iSl 


RESPONSE  RATIONALE; 


5E5EQ!i5g-5S25Si _ 

(COMPLETELY  DISAGREE  -  1,2,3,4,S,6  -  COMPLETELY  AGREE) 


QUESTION  DATA  SHEET 


Quaatlon  Nuxbar  SPH(PI>  -  008 

QUSSII9!il  Opttr«tlon«l  TMt  «fid  Evaluation  (0T6E>  organization 
axtarnal  intarfacaa  ara  adaquata. 


Proeuraaant 

gXPVAWATION ;  Tha  0T6E  ia  aanagad  by  AFOTEC.  0T6S  ia  conductad  by 
AFOTEC  and/or  participating  coaaanda.  Intarfacaa  with  naarly  all 
program  organization  alaaanta  ia  poaaibla  dapanding  upon  tha  ayatam 
baing  acquirad.  Soaa  of  tha  functiona  of  thaaa  intarfacaa  ara  to 
coaaunicata  an  aatiaata  of  tha  ayataa'a  oparational  affactivanaaa  and 
auitability#  Idantify  oparational  daficianciaa,  racoaaand  ehangaa, 
and  idantify  ayataa/aoftwara  charactariatica  and  daficianciaa  which 
can  aigniflcantly  impact  aupport  coata.  Saa  AFR  B0-14»  AFR  55-43*  and 
AFOTECR  55-1  for  mora  information. 


GLOSSARY; 

QT AS.Qygf p^jyation .  Tha  organization  alamanta  (primarily  AFOTEC 
and  participating  comaanda)  raaponalbla  for  tha  aatiaation  of  a 
ayataa'a  oparational  affactivanaaa  and  auitability*  idantification  of 
any  oparational  and  aupport  daficianciaa*  and  idantification  of  any 
naad  for  aodificationa. 


55§ESl!SS_I22IS2SIISiS5i 


RESPgMSE_SC0RE2 _ 

"<c5MPLEfiLY“DISAGREE  ■  1*2*3*4*5*6  ■  COMPLETELY  AGREE) 
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QUESTION  DATA  SHEET 


QUESTION;  Th«  Conputsr  R««ourc«s 
intarfacM  %vm  adcquata. 


QuMtion  Nu»b«r  SPH(PI)  -  009 


Working  Group  (CRWG)  oxtornal 


ACTIVITY<g>  I  ProcuroBonb 

ooch  ayatoa  utilizing  coaputor  roaourcoa,  a  CRWG 
■uat  ba  aatabliahad  iaaodiataiy  a£tar  NilMtona  I  to  aid  in  tha 
nanaganant  of  tha  ayataa'a  eoaputar  raaoureaa  and  in  tha  davalopaant 
of  tha  CRLCHP<CRISP> .  Tha  purpoaa  of  tha  CRWG  ia  to  aaaiat  tha 
program  managar  in  initiating  activitiaa  that  ara  praraquiaitaa  to 
davalopaant  and  aupport  of  eoaputar  raaotireaa.  Tha  CRWG  ahould  also 
aaaiat  in  anauring  that  eoaputar  raaoureaa  eoaply  with  aatabliahad 
polleyf  proeaduraa#  plana «  and  atandarda.  Tha  CRWG  ahould  ineluda 
rapraaantativaa  of  tha  proeuraaant  aetivity*  oparation  aupport 
aetivity,  and  alao  othar  organlzationa  that  hava  baan  aaaignad 
raaponaibilitiaa  for  aoftwara  davalopaant^  taating^  training*  and 
aupport . 

skQK&sn 

CPWG.  eoaputar  Raaoureaa  Working  Group  ia  ehairad  by  tha  program 
offiea  and  eonaiata  of  rapraaantativaa  of  tha  partieipating  eoaaan^^^ 
and  taat  organlzationa.  Primary  funetion  ia  to  produea  CRLCHP (CRISP )V 


5SS£2U§i-Ilf5I5USII21!Si 


RE^PQNgg  .RAT^gWALg ; 


vCCKPLZTZL'/  .DISAGREE  »  *  COMPLETELY  AGREE) 
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QUESTION  DATA  SHEET 


QuMtion  Number  SPH<PI>  -  010 

flUSSUSlil  Tha  T«st  Planning  Working  Group  (TPWG>  oxtornal  intorfacoa 

ar a  adaquata . 


All 

t  TPWG  aaaiata  tha  program  nanagar  on  taat  aattara. 
Thia  aaaiatanca  includaa:  dafining  raaponaibilitiaa  and  ralationahipa 
among  taat  program  participanta;  aatabliahing  taat  objactivas; 
praparation  of  tha  TEMP;  idantifying  taat  raaourcaa;  praparing  taat 
portiona  of  RFPa,  SOWa,  and  ralatad  contractual  documanta;  and  acting 
aa  a  forum  for  aurfaeing  and  raaolving  taat-ralatad  iaauaa. 


glossary; 

fgWg^  Tha  Taat  Planning  Working  Group  ia  chairad  by  tha 
implamanting  command  with  rapraaantativaa  from  tha  uaing  and 
aupporting  commanda*  and  tha  taat  organizationa  (DTAE,0T6E>,  and 
whara  appropriata,  tha  davalopmant  contractor  and  aubcontractora . 


RESPONSE  instructions; 


RESPONSE  rationale: 


RE3P0MSS_SC0gE;  _ 

“cSkPlItSLY  5iS agree  ■  1,2,3,4,S,6  ■  COMPLETELY  AGREE) 
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QUESTION  DATA  SHEET 


Quastion  Nujibar  SPHCPI) 


fiUSSIISlil  Intarfaca  Control  Working  Group  CICUG)  oxtarnal 

Intarfacos  mrm  adaquata. 


ACTIVITY (S) :  All 

gXPlf ANAT^ON ;  Tha  procuraaant  activity  should  aatabllah  an  .  ICWG  to 
addraaa~aystan/aubayataa  intarfaca  raqulramanta*  including  thoaa  that 
a£fact  computar  raaourcaa.  Early  In  tha  syataa  acquisition  procass, 
tha  ICWG  supports  tha  procuraaant  activity  In  dafinisg  currant  and 
proposed  computer  software  and  hardware  Interfaces.  Tha  Intarfaca 
description  should  Include  quatitatlva  data  needed  to  accurately 
define  Interfaces.  Interoperability  requirements  should  be  Included 
in  the  Interface  definition.  The  procurement »  development  contractor, 
and  operation  support  activities  should  provide  computer  resource 
inputs  to  the  ICWG  to  ensure  that  system  Interfaces  adequately 
reflect  software  and  hardware  characteristics. 


G^SgSASYj, 


V-/ 


:^2S?0MSS  RATIONALE ; 


SC0R2 

a 

• 
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TELY  D 

iSAGREE  « 

CONPLSTELY 
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QUESTION  DATA  SHEET 


QuMtlon  Number  SPMCPI)  -  012 

OUESTigx;  Th«  Ind«p«nd«nt  Verification  and  Validation  (IVfcV)  agency 
external  intarfacee  are  adequate. 


ACTIVITY  (S) ;  All 

procurement  activity  ehould  determine  IV&V 
requirementa.  The  CRWG  will  aaaeaa  the  need  for  IV&V  and  provide 
recomnendationa  to  the  procurement  activity.  The  determination  ehould 
coneider  the  criticality  of  the  ayatem  miaaion^  the  riak  aaaociated 
with  the  development,  and  the  level  of  aoftware  complexity.  The 
procurenent  activity  ehould  define  the  acope  and  timing  of  the  IV&V 
and  dwvalop  a  plan  for  the  IV&V  effort.  The  IV&V  deciaiona  ahould  be 
documented  in  the  CRLCH?.  If  IV&V  ia  to  be  included  in  the  software 
development,  it  muat  be  initiated  not  later  than  the  award  of  the 
development  contract.  Procurement  activitiea  for  the  IV&V  effort 
should  be  completed  in  advance  of  the  Software  Specification  Review 
to  allow  verification  of  the  aoftware  requirementa  bafore  tha  review 
is  conducted.  The  procurement  activity  ahall:  control  tha  intarface 
betwaen  the  IV&V  agency  and  tha  davalopmant  contractor;  provlda  the 
IV&V  agency  with  eopiea  of  the  appropriate  development 
specif icationa,  daaign  documenta,  liatinga,  and  discrapaneies  found 
by  the  IVaV  agency. 


ir'STPVCTlOh'S : 


► 


RESPONSE  RATIONALE: 


0 


SCCR 


I  A1-12S 


-  .A' 


QUESTION  DATA  SHEET 

Qu«s:tion  Nuaibor  SPMCPD 

?VH5**3ny :  Th«  so5'twar«  configuration  aianagitnant  Intarfacas  anor.g  •■•11 
actlvitiSa'  aanaganant  conponanta  for  tha  aubjact,  ayatan  ar^ 
afaquata. 


ACTIVITY All 

A  NATION:  Each  of  tha  aetivitiaa  has  raaponaibilitiaa  for 
configuration  managaaant  which  raquira  intarfaca  conaunication . 
Aaalgnmant  of  idantif icatlon  nuabara  ia  a  procuramant  and  oparation 
aupport  CAFLO  raaponaibility  dapandant  upon  tha  '  davalopaar.t 
contractor  formal  raquaat  for  auch  nunbara.  Tha  procuramant  activity 
r.iintaina  tha  formal  baaalinaa  (Functional,  Allocatad,  Product)  wh-la 
d«»vialopmant  contractor  maintaina  Davalopmahtal  Eaaalinaa  up 
-:.v«r,ry  of  tha  Product  Baaalina.  Changaa,  rafinamanta,  and  ao  forth 
rt.^t  ba  communicatad.  Tha  procuramant  and  davalopmant  contractor  mu.'<t 
crordinata  tranafar  of  configuration  managanant  raaponaibility  t-v 
•:*fln*»d  in  tha  CPhCH?  (CSISP,  0/S  CXP)  with  tha  oparation  auppcrt 
.i«;t  ivity . 


T-r.nssAEY: 


,  js : 


LSQu2a_2AT’gNAlg£ 


CUSSTION  DATA  SHS2T 


CuMtlon  Number  SPH(PI>  *  014 


CUgg'TTC.VL  Tha  aoftwarm  quality  acmuranem  aanagamant  Intarfacaa  among 
all  activitiaa'  mar.aganant  componanta  for  tha  aubjaet  ayatam  ara 
adaquata . 


^STsYffTy.^S)  :  All 


S<EtA2J^IIS3i  1*ha  ova  rail  aoftwara  quality  program  for  tha  computer 
aoftwara  development  cycle  ahould  be  defined  by  tha  procurement 
activity  and  operation  aupport  activity  through  contractual 
raquiremanta.  Raaponaibility  for  aaaaaalng  computer  aoftwara  producta 
and  rolatad  procaduraa  may  be  aaalgnad  to  more  than  one  organization. 
It  may  be  appropriate  for  an  independent  organization#  auch  aa  an 
IVkV  organizatin#  that  la  aubjact  to  neither  financial  nor  managerial 
control  by  the  development  contractor#  to  perform  certain  of  theae 
1  j  laaanantd . 


a.  laV  • 


IM£TP?.*CTICNS  I 


SSSPONgS  ,gAT;gNA^,E!_ 


’“C0MPL2TiLY"5lSAGPii”*”l.2.3,4.5.6  C0MPLST2LY  ACRE2> 
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QUESTION  DATA  SHEET 


QuMtlon  Viuib«r  SPH(PZ)  *■  019 

QUESTION;  Th«  contract  nanafonant  intarfacaa  aaong  all  actlvitlaa' 
nanaganant  coaponanta  for  tha  aub^act  ayatan  ara  adaquata. 

ASIIVimgll  All 

Contactual  proviaiona  ahould  raflact  tha  Govarnaant'a 
raquiraaanta  for  righta  to  tha  coaputar  aoft%Mra  and  aaaoclatad 
docuaantation .  Davalopaant  contractor  llaitad-righta  aoftwara  to  bo 
laaad  in  tha  parforaanca  of  tha  contract  or  to  ba  dal  Ivor  ad  undar  tha 
contract  ahould  ba  Idantifiad.  Pra-dataralnation  agraaaanta  ahould  ba 
Includad  in  tha  contract  to  anabla  tha  Govarnaant  to  aubaaquantly 
acquira  additional  raquirad  righta.  Bacauaa  coaputar  raaourcoa 
(including  coaputar  aoftwara)  aay  ba  davalopad  undar  a  aubcontract  to 
a  priaa  contractor,  tha  procuraaant  activity  ahould  anaura  that  all 
appropriata  contractual  raquiraaanta  laviad  on  tha  priaa  davalopaant 
contractor  ara  paaaad  to  tha  aubcontractor .  Tha  procuraaant  activity 
ahould  anaura  that  tha  contract  aakaa  tha  aubcontractor  raaponaibla 
for  tha  integrity  of  aubcontractad  producta  and  aakaa  tha  priaa 
davalopaant  contractor  raaponaibla  for  dalivaray  of  an  accaptabla 
product  undar  tha  contract. 

Sl.S55AS2i 


3gg2Sl!SS_iNST3UCTigNSi 


RESPONSE  RATIONALE; 


Rg5EQN5g_5SORE;  _ 

■■  .ZCriPLITlL'f  "DZSAijP.SS  » 


•  # 
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QUESTION  DATA  SHEET 


QuMtlon  Number  SPM (PI >  -  03.6 

QUESTION;.  The  Interaervice  externel'  Interfecee  with  all  actlvltlea' 
management  componenta  are  adequate. 


ftsuvimsii  All 

Before  each  ayatem  milaatone*  Interaarvlelng  potential 
ahould  be  reviewed  and  the  management  and  life  cycle  coat 
Impllcatlona  of  major  aoftwara  aupport  optlona  ahould  be  analyzed. 
Thla  analyala  ahould  alao  conaldar  Impact  on  operational  needa, 
configuration  management*  and  ayatem  Integration.  For  Interaervice 
ayatama*  the  CHUG  and  ICWG  ahall  ba  Interaervice  groupa  that  Include 
repreaentatlvea  from  the  cognizant  participating  organization 
alamanta  (commanda*  aganclaa,  ate. > .  The  Interaervice  working  groupa 
ahould  anaura  that  analyala  la  performed  to  determine  the  optimum 
aupport  approach*  thla  analyala  la  documented*  and  recommendatlona 
are  made  to  the  procurement  activity  eonearnlng  the  aupport  approach. 


gyeSSARY ; 


Rg^PONSE. INgTRUCTIONS ; 

*  A/6:  There  are  no  joint  aervlee  requlrementa  for  the  ayatem 
or  Ita  embedded  aoftware. 


555BSl!5f-5AI19!!A|sIi 


RESPONSE  SCORE;  _ 

"(COMPLETELY’SiSAGRii  ■  1*2*3*4*S*6  ■  COMPLETELY  AGREE) 


A1-12S 


SOFTWARE  CONFIGURATION  NANACEMENT  IDENTIFICATION 


m 

Th«  quMtlona  8CM( ID >-001  through  SCM<I0>-021  addroaa  laauos  of 
•oftwaro  configuration  identification  for  tha  procuraaant, 
davalopaant  contractor*  and  operation  aupport  activitiaa. 
Configuration  identification  ia  eatabliahad  in  tha  fora  of  tachnical 
documentation  that  becoaea  more  datailad  aa  davalopaant  proceeda  and 
more  refined  aa  tha  final  development  producta  are  evolved  during 
poet  deployment  aupport.  Three  atagee  of  configuration  idantification 
are  generally  employed  during  the  aoftware  ayatem'a  life  cycle: 

<1)  Functional  Configuration  Identification 
<2>  Allocated  Configxiration  Identification 
(3)  Product  Configuration  Identification 

In  addition*  tha  davelopment  contractor  activity  haa  intarnal 
itarationa  of  the  identificationa  callc<-d  Davelopment  Identif icationa 
which  are  controlled  by  the  neceaaary  Intarnal  aoftwara  configuration 

management  proceaa. 


Theme  Identificationa  corraapond  to  the  three  ayatem  development 

baaelinea: 


.<1)  Functional  Baaeline 
(2)  Allocated  Baaeline 
<3)  Product  Baaeline 


and*  the  appropriate  development  contractor  Davelopment  Baaelinea. 
The  Identificationa  become  Baaelinea  when  the  procurement  activity 
approvaa  the  Identificationa  and  puta  the  configuration 
idantification  under  ita  contractual  configuration  control  ayatam. 
Identification  ia  uaed  for  viaibility  and  baaelinea  ara  uaad  for 
control . 


The  term  ‘*icentif  ication'*  alao  haa  an  important  aacondary 
meaning  aa  a  document  or  aet  of  documanta  that  dafinaa  tha 
configuration  of  an  item.  In  thia  aenae  it  rapreaenta  one  or  more 
material  thinga  (documanta) . 


# 


Al-130 


question  data  sheet 


QuMtlon  Nuabar  SCMdO)  -  001 

flUSSIIQlil  Th«  procur«a«nt  policy*  standards*  and  convantlons  appliad 
to  tha  Idantlfication  of  software  configuration  itaaa  ara  adaquata. 


ACTIVITXiSlI  Procuraaant  . 


EXPLAWATIOyS;  Identification  of  computer  software  configured  items 
and  procadurea  for  assigning  identification  numbera/namea  is 
described  in  AFR  800-21.  It  is  the  responsibility  of  the  procurement 
activity  to  assure  that  proper  policy*  standards*  and  conventions  ara 
required  for  the  naming  of  configured  items.  Directive  and  guidance 
documents  include  AFR  6S-3  and  AFR  800-14 (Vol  II >.  Contractor 
compliance  documents  which  could  be  required  include  HIL-STD-480A* 
HIL-STD-482A*  HIL-ST0-483A*  and  HIL-STD-490A*  and  DoD-STD  2167. 


GLOSSARY; 

jl^iQO:,_|tems_8  Software  a  laments  which  ara 
designated  for  configuration  control  by  the  contractual  raquiremanta . 

ydentiff ieg^4gqt  The  official  charactar/numeric  identifier  of  a 
configured  item  and  its  functional  purpose  and  relationship  with 
other  configured  items  for  purposes  of  configuration  management. 


5S§E91f§S-Il!§ISiiSII21!5i 


BKE9N2I.5CQRii _ 

(COMPLETELY  DISAGREE  «  1*2*3*4*9*6  -  COMPLETELY  AGREE) 
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QUESTION  DATA  SHEET 


QuMtlon  Niiab«r  SON  (ID)  **  002 

OU^yf ^ON ;  Th«  proeur*n«nt  Identification  of  dallvarabla  aoftwara 
configuration  Itaaa  la  adequate. 


I  ACTIVITY <S> ;  Procurement 

EXPLANATIONS;  Frequently  aoftware  Iteaa  are  not  required  to  be 
delivered.  The  CDRL  la  the  baele  for  delivery.  The  CORLa  ahould 
carefully  Identify  all  operational »  teat»  and  aupport  aoftware 
dellverablea.  Included  with  thla  identification  la  the  fora  of  the 
deliverable  (e.g.#  document*  aource*  object*  load  nodule)*  and  tha 
medium  on  which  the  deliverable  will  be  produced.  Include  In  tha 
contract,  all  aoftwara  Including  firmware  and  proprietary  Itema  that 
are  required  to  coat  effectively  uae*  operate*  or  modify  the  ayatam 
over  Ita  life  cycle.  If  It  la  not  coat  affective  to  acquire  a 

[  aoftware  Item*  Include  an  option  to  acquire  It  later. 

i 

GLOSSARY: 

I  Deliverable  Software*  Identified  by  CDRLa. 

1  ■  Required  to  operate  the  ayatem. 

Tft^  Sgfttwgre.  Uaed  to  analyse  or  teat  ayatem  and  component 
performance . 

SiiSSSIi^SSi^HMSSj.  VMd  generelly  to  develop .  or  maintain  othe. 
aoftware.  Includea  ayatem  aoftware  much  aa  operating  ayatema  and 
compllera. 

I 


!■ 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 


CCr.PLITILV  -TSaGSES  »  -.2.3. -4. 3, 3  «  CCMPLZTcL?  AGREE) 
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QUESTION  DATA  SHEET 

Quest  ion  Nusbsr  SCM(ID)  003 

QUESTION:  The  proeuressnt  activity  identification  of  the  software 
configuration  baselines  is  adequate. 


dSXiyilXlSll  Procurement 

gXPLAyATIO^^  *  software  configuration  baselines  include  the 
Functional  Baseline,  Allocated  Baseline,  and  Product  Baseline.  When 
the  procurement  activity  approves  the  development  Configuration 
Identification  for  each  of  these  baselines,  then  the  Identification 
becomes  the  corresponding  Baseline  and  is  put  under  the  procurement 
activity  configuration  control.  This  requires  the  procurement 
activity  to  have  adequate  identification  capability  to  maintain  such 
configuration  control. 


GLOSSARY t 


S5S£Qi!SS.INSIRugiiSNSi 


RESPONSE  RATIONALE: 


CCeSPLETSLY  2I5AGP.22  »  1.2, 3, 4, 5,5  »  CCNPLETILY  AGREE) 
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QUESTION  DATA  SHEET 


QuMtlon  Nuabsr  SCM(ZO)  -  004 

QUESTION r  Thm  mytmm/mmgmmnt  •pseifleation  adaquataly  Idantlfiaa 
•lamanta  of  tha  aoftwara  functional  baaallna. 


ACTIVITY <S) ;  Procuraaant 

gXpLAyATIONS ;  Tha  ayataa/aaqnant  apaeification  la  a  foraally 
controllad  aoftwara  itaa  of  a  ayataa  procuraaant.  Thia  apaeification 
dafinaa  tha  Functional  Configuration  Zdantifieation  of  tha  aoftwara. 
Tha  firat  varaion  ia  uaually  praparad  by  tha  procuraaant  activity  and 
baeoaaa  a  living  docuaant  of  tha  ayataa/aagaant  parforaanca-  oriantad 
raqulraaanta .  Whan  it  ia  approvad  by  tha  procuraaant  activity,  it  la 
**baaalinad**  and  eoaaa  undar  configuration  control  aa  tha  Functional 
Baaallna.  Tha  Functional  Baaallna  ahould  ba  availabla  at  Hilaatona  I, 
prior  to  Oaaonatration  and  Validation.  It  ia  critical  that  thia 
apaeification  raflact  aa  coaplata  a  parapactiva  on  tha  functional 
aapacta  of  tha  ayataa  aa  poaaibla.  It  ia  alao  critical  that  thia 
apaeification  natura  aa  aarly  aa  poaaibla  to  nlniaiza  parturbationa 
on  tha  raat  of  tha  ayataa  baaalinaa. 


S&SSSaryi 


SSSPCNSE^INSTRUCT^ONSi 


RgSf9NSE.  RATIONALE; 


5§§£2I*ss.ScgREj. 


4.2.5 


:C2?LETELy  agree; 


QUESTION  DATA  SHEET 


QuMtion  Nu»b«r  SCH(ID>  ■*  005 

^UgSTIONr  Th«  p«rfor»«ne«  r*quir*»«nt  specifications  sdsqustsl/ 
identify  slsssnts  of  the  software  el located  baseline. 


ACTIVITY <S?1  Procureaent 

\  The  performance  requirement  specifications  are  the 
descriptions  of  how  the  Functional  Baseline  is  allocated  ' into  CSCIs 
(Computer  Software  Configuration  Items)  and  HWCIs  (Hardware 
Configuration  Items) .  These  specifications  include  preliminary 
requirement  documents#  and  become  living  documents.  When  they  are 
approved  by  the  procurement  activity  they  ere  "baselined**  as  the 
Allocated  Baseline.  The  Allocated  Baseline  should  be  available  at 
Milestone  II#  prior  to  full  scale  development.  It  is  critical  that 
these  specifications  reflect  as  complete  a  perspective  on  the 
detailed  function  allocation#  test#  and  interface  aspects  of  the 
ayatam  as  possible.  It  is  also  critical  that  these  specifications 
mature  as  early  as  possible  to  minimize  full  scale  development 
perturbations . 

CLOSBARY ! 


?SS?Cy.S2  INSTRUCTIONS: 


•  RESPONSE^ RAT ION Ays: 


_ 
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I 


QUESTION  DATA  SHEET 


Question  Nuaber  SCM(IO>  **  006 

QUSSJ^Qy ;  The  iaplementetion  speci£lc«tlone  edaquately  Idantlfy 
elamanta  of  the  aoftware  product  baaellna. 


ACTIVITY <S) ;  Developaant  Contractor 

Tha  laplaa  entation  apeeificatlona  ara  tha  daacrlptlon 
of  how  the  Allocated  Baseline  has  been  iapleaantad  aa  CSCZa  (Computer 
Software  Configuration  Iteaa)  and  HWCZs  (Hardware  Configuration 
Ztaas) .  These  isplesentation  apacifications  include  tha  “aa  built" 
detailed  design  documents,  and  bacons  living  docunanta.  Whan  they  ara 
approved  by  the  procurement  activity  they  are  "baaalinad"  and  bacons 
tha  Product  Baseline.  It  ia  critical  that  thaae  apacifications 
reflect  as  conplata  a  perspective  on  tha  detailed  design  and  coded 
aapacta  of  the  system  aa  possible.  It  ia  also  critical  that  these 
specifications  nature  aa  early  as  poaaibla  to  facilitate  tha 
transfer,  operation,  and  support  of  the  software. 

SMSSARY: 


SI522!!5S_I-!5ISyS2I2!fSi 


REgP0N§S_gATI0NALE: 


SiSPONSE^gCOggi 


I2ACRSE  »  1.2.3, 4.5. 3  ■  CCrtPLSTELV  AGREE: 
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QUESTION  DATA  SHEET 


QuMtion  Nuaibar  SCIldD)  -  007 

oyESTION;  Th«  Identifier  eharecterietice  for  eoftwere  configuration 

item  naaea  are  edeguete. 


i^9TIVITY(S) ;  Developeent  Contractor 

SiSS^&H^IIQSSJ.  The  identifier  character latice  include:  uniqueneaa, 
retrievability,  traceability#  pronouncibility#  variability# 
functional  aignif icance#  and  coapaetneae. 


GLOSSARY: 

e 


RSSPONSS  INSTRUCTIONS; 


SSSPONSE^RATigNAtli 


"rc5ii?LiTiLY"3lSAGRii'”»”l  ,2,3,4.5.S  «  COMPLETELY  AGREE) 


QUESTION  DATA  SHEET 


QuMtlon  Number  SCM(IO)  -  008 

flllSSXISlil  development  contractor  internal  identifier  naming 

etandarda/conventiona  eetiafy  contractual  regulationa. 


/ACTIVITY <S) ;  Development  Contractor 

;  The  development  contractor  ia  (ahould  ba)  raquired  to 
follow  Air  Forca  ragulationa  on  Computar  Program  Idantif ication 
Number  (CPZN)  aaaignmenta.  Air  Force  directive  guidance  ia  found  in 
AFR  6S-3  and  AFR  600-14  <Vol  IZ)»  along  with  other  documenta.  The 
development  contractor  compliance  documenta  are  DoD-8TD-4dOA, 
HZL-STD-462A,  HZL-8TD-4d3A,  and  MIL-8TD-4SOA.  MIL-STD-490A  ia  the  key 
regulation  for  document  identifier  requirementa*  and  MIL  STD  483A  ia 
the  key  document  for  CSCZ  identifier  requirementa . 


GLOSSARY: 


RESPONSE  instructions: 


I  RESPONSE  RATIONALE: 


I 

RE3P0NSE_SCgREi. 

”“cC???iIiTiLV“5  IS  AGREE  »  1.2. 3. 4. 5. 5  »  COMPLETELY  AGREE) 
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6MMMWBiBgvmvwyanQnQffQnQiniwyii«rKinMQniinuiKM»aarwMiununiintuT^^ 


QUESTION  DATA  SHEET 


QuMtlon  Numbttr  SCIKID)  -  009 

QySSISSlil  Davalopmsnt  contractor  Idantlf  Icatlon  standards  and 
convantlons  can  ba  tranaltlonad  to  operation  support  standards  and 
convantlona. 

ACTIVITY <3) *  Davalopaant  Contractor 

In  order  for  computer  raaourcaa  to  ba  aaoothly 
tranaltionad  from  the  davalopaant  contractor  to  tha  oparatlon  support 
activity*  tha  configuration  identification  standards  and  conventions 
must  ba  coapatibla.  Aa  sore  autoaatad  tools  are  used*  this 
raquiramant  for  compatibility  will  ba  even  stronger.  Evidanca.  of  tha 
standards  and  transition  strategy  should  ba  in  tha  CRLCMP  (CRISP  and 
0/S  eSP)  aa  wall  as  tha  davalopaant  contractor  software  conf iguration 
nan^gamant  plan. 


GLOSSARY: 


r^sspcuss  _iN3TRUc?:gM3; 

?/i:  No  standards  exist  for  tha  development  contractor  activity 
and/or  the  operation  support  activity 


RgSPgNSE  RATIONALE: 


RES2C2IS|_SCORSi _ 

""cC?f?Li?iLy”5lSAG?SH  »”l  .  0 . 3 . 4 . 5 . 5  «  COMPLETELY  AGREE ) 
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QUESTION  DATA  SHEET 


QuMtion  Number  SCH(ID>  -*  010 

9ygynQN:,  D«v«lopment  contractor  deliverable  configuration  itama  are 
named  to^daquately  Identify  multiple  varaiona  and  variationa. 


A<fTIVITX<?>  *  Development  Contractor 

Tti*  minimua  requiramant  ia  for  the  name  to  provide  for 
diaerimlnation  of  veraione.  If  aoftware  muat  be  configured 
apecifically  for  teat  purpoeeap  multiple  aiteap  and  ao  forth*  it  will 
be  neceeaary  for  the  name  to  diatinguiah  much  variationa  of  each 
veraion . 


GLp^^lRY,: 

Verf ion«  Baaeline  raleeee  of  a  configuration  controlled  item. 

One  of  at  leeat  two  phyaieal  eonfigurationa  of  the 
aame  veraion  of  a  eonf igureblon  controlled  item.  Variationa  of  a 
.veraion  emiat  to  support  multiple  aervice  requirementa  aa  well 
miaaion  apecific  eonfigurationa  (teat*  operational  miaaion  acanarioW 
alternate  embedded  computer  ayatema*  etc.). 


5£SES2!§S-IS§I5HSII21iSi 


RESPONSE  RATIONALE; 


C COMPLETELY  OISAGRSS 


•  1,2.3.4,S.<5  ■  COMPLETELY  AGREE) 


QUESTION  DATA  SHEET 


QuMtion  Number  SCH(IO)  ~  Oil 

Dmvelopm  •nt  contractor  Idmntiflcmtion  procodurms  are 
structured  to  permit  easy  addition#  deletion#  or  modification  or 
configured  itema  at  any  hierarchical  level. 

ACTIVITy,^??.i  Development  Contractor 

T  identification  procedures  should  be  specif iad  in 
the  Software  Configuration  Kenegenent  Plan  or  perhaps  a  eat  of 
procedures  to  implement  portions  of  the  Plan.  Hierarchical  levels  are 
C8CZ#  component#  module#  end  routine.  Such  procedures  are  essential 
for  adequate  software  supportsbility. 


GLOSSARY; 


5=SSSl!SI-52!2ISySII2N5i 

F/l:  No  such  procedures  are  documentad  or  are  totally 
inadequate 


5SSP22!2S.SATigifAt5i 


'7c5MPLiTiLy“o:SAGiii~”l.2.3.4.S.3  «  CCMPLSTSLY  AGREE) 


Al-141 


QUESTION  DATA  SHEET  ^ 

QuMtlon  Number  SCH(IO)  -  012 


QUESTION;  Development  contreetor  Identification  procedurea  for 
addition,  deletion,  and  modification  of  configured  Iteme  are  being 
followed. 

ASXiyilXiSll  Development  Contreetor 

It  ahould  be  poeelble  to  determine  whether 
identification  procedurea  era  being  followed  through  the  atandard 
management  reporting  requlrementa. 


GLOSSARY  t 


1^ 


SS§52JfSI-Il!§ISySIISU5i 

F/l;  No  much  procedurea  exlat  or  are  being  totally  ignored 


RggPQNg^  RAT^QWAIfE; 


RESPONSE  SCORE:  _ 


»  1.2.3,4,5,S  »  C0KPL27SLY  AGREE: 


QUESTION  DATA  SHEET 

Qucatlon  Nuabar  SCH(IO>  -  013 

QUESTION!  Th«  phyaieal  aadiua  o£  conflgurad  itaaa  la  adaquataly 
daaerlbad  by  tha  davalopa^nt  contractor  software  eoaponant/itan 
Idantlflcatlon  aehaaa. 

Davalopnant  Contractor 

g^pVANATTOWS!  Tha  phyaieal  nadltaa  (a. 9.,  taps,  disk*  aaaory 
eoaponanta)  of  configured  Itaaa  ia  raquirad  to  naat  Govarnaant 
standards.  Part  of  thoaa  standards  .  addraaa  idantif ication 
naaaa/nuabara.  It  should  ba  poaalbla  to  traea  tha  aadlua  of  a 
configured  itaa  from  its  daacrlptiva  label /nans  and  any 
diatinguiahing  aapacta  of  tha  aadiua  (a.g.*  working/aaatar  tapa, 
aaquantial  voluaa  number  for  multi** volume  storage  itama)  . 


StSSSASXi 

Disk,  tape*  card  deck,  firmware,  read-only  mamofy,  ate. 


e 


RESPONSE^INgTRyCTIONS^ 


gSgPONSS  R ATIONAyg : 


QUESTION  DATA  SHEET 


.*> 


QuMtlon  NualMr  SCH(IO>  **  01 


9V^STI9N;  Th«  davalopusnt  contractor  aoftwara  idontifiora  adoquataly 
diatinquiah  aaong  dlffarant  atataa  <a.$|^.»  aourca*  objact,  load*  cora 
laagaa*  llatlnga)  of  tha  aoftwara. 

^CJXVITV<S)  ;  Davalopaant  Contractor 

pyPU^NATIONS:  Tha  idantiflar  ahould  diatingulah  which  atata  tha 
ao£twara  itaa  ia.  For  axaapla*  a  diatlngulahlng  a\i£flx  might  ba 
attached  to  tha  aoftwara  itaaidantlf iar*  aueh  aa:  **.prg***  ‘*.txt‘'> 

••.cmd”,  ".dat”. 


GltOSSARY; 


m 


ESSEQNSi.lNSIBUSIIS!iSl 


RESPONSE  RATIONALE; 


“CKPLiTiLy*5lSAGiiF**lp2,3,4,5mS  «  COKPLSTELY  AGREE) 


QUESTION  DATA  SHEET 


OuMtlon  NuMb«r  SCH(TO>  -  OIS 

fiSISSIIQlIl  davalopmant  contractor  software  change  control  form 
Idantif iara  are  adaquat*. 

ACTIVITY <S> :  DaValopmant  Contractor 

gypy^H^TIOKS ;  The  change  control  form  Idantifiara  should  meat 
raquiraaanta  of  applicable  govammant  standards  and  provide 
saquantial  identification  auitabla  for  logging,  filing,  rafaranca, 
and  retrieval. 


SLOSSARYj. 


e 


RZ3PQM3S  INSTRUCTIONS: 


REgPQMJg^RATJQMALEi 


R5SPCNSS  .SCORE :  _ 

“cOMPLETiLY”  IS  AGREE  ■  1.2.2.4.S.S  ■  COMPLETELY  AGREE) 


5 


nnwi  AjaunnKMUiuai. 


A1-14S 


QUESTION  DATA  SHEET 


QuMtion  Nu»b«r  SON  (ID)  -  01 


QUg?TIQN ;  Subcontractor  configuration  itom  idantlflcatlon  practicaa 
ara  aonitorad  by  tha  davalopaant  contractor. 


ACTIVITY I  Davalopaant  Contractor 

BCPVANATIOWg.;  Zf  thara  la  a  aubcontraetor»  It  will  ba  nacaaaary  that 
tha  davalopaant  contractor  raqulra  configuration  idantif  ication 
practicaa  aiailar  to  thoaa  raquirad  by  tha  procuraaant  activity.  If 
thia  ia  not  dona#  than  tha  davalopaant  contractor  will  ba  raquirad  to 
retrofit  tha  idantif  ication  achaaa  of  tha  aubcontractor .  Tha 
idantif  ication  practicaa  of  tha  aubcontractor  auat  ba  earafully 
aonitorad  to  aaaura  coapaiibility. 


9V9SSARY : 


3I5S22f§i-Is!§IS2SII2!!5i 

A/6:  No  aubcontractora  ara  involvad  with  producing  aoftwara 
configuration  itama  for  tha  davalopaant  contractor. 


RESPONSE  _  RATIQNAI,S ! 


'Tc5M?iIZTiLY”5lSAGRii”»'’l,2,3,4,5.S  «  COMPLETELY  AGREE) 


Al-146 


QUESTION  DATA  SHEET 


OuMtlon  Nuabar  SCHdD)  *  017 

QUESTION:  Th«  documantatlon  which  collectively  identiflee  the  content 
of  e  conf Iguretlon  Itee  le  adequate. 


Activity  <8> :  Oevalopeent  Contractor 

'The  docueentation  eight  include  a  varalon  deacriptlon 
docuaant,  or  a  aoftware  configuration  index.  The  veraion  deacriptlon 
docuaant  uauelly  identiflee  changea  to  a  baaeline  product.  The 
configuration  index  ia  a  repraaentetion  of  the  full  baaeline  product 
conponente  (e.g.,  in  a  hierarchical  chart)  ahowing  component 
ralationehlpa . 


GLOSSARY: 


RSSPgNgg  INSTRUCTION^; 


3l2EQiISS-S&Ia25&kii 


'•r:f:^^?L^TSLY  0  IS  ACRES  «  1,2,3.4,5,S  «  CCKPLETSLY  AGPSS) 


QUESTION  DATA  SHEET 


QuMl^lon  NuBb«r  SCMCZD)  '  Old 


OUEyn<?N  f  Softwars  conSigurad  itaas  which  laplanant  aafaty  provlalona 
ara  adaqualialy  IdantlSiad. 


Oavalopaant  Contractor 

;  Configurad  itana  which  iaplaaant  aafaty  proviaiona  ara 
fraquantly  controlled  by  aoftwara.  Thia  aoftwara  auat  ba  adaguataly 
idantifiad  aa  affacting  aafaty.  Safaby  proviaiona  ara  cloaaly  ralatad 
to  tha  raliability  of  aiaaion  critical  coaponanta,  aafaty  of  aiaaion 
paraonnal*  nuclear  affacta*  and  ao  forth. 


GLOSSARY: 


3gSPONSE_INSTRyCTjEON§i 

A/6:  Thara  ia  no  raquiranant  for  aafaty  proviaiona  to  bi 
iaplanantad  or  eontrolj,ad  by  aoftwara 


R^PONgS  RATIONALE  I 


Bi§P22!§I-5SQBiI 


•••vCT  *r  ’  *•  -  si  *  ^  m  'T'rTT  V 


QUESTION  DATA  SHEET 


QuM^lon  Nuiibar  SCH(IO>  -  019 

QUESTION ;  Software  conflgurad  Itaas  which  iaplcmcnb 
eoiiput«r/eomnunle«tions  aceurity  provlaiona  ar*  adaquataly 
Idantlflad. 


J  Davalopaant  Contractor 

Softwara  which  iaplaaanta  coaputar/coaaunication 
aacurity  ia  particularly  important.  Any  auch  aoftwara  itama  muat  ba 
adaquataly  idantifiad  aa  part  of  tha  truatad  coaputar  baaa.  If  tha 
configurad  aoftwara  itan(a)  ara  thaaaalvaa  claaaifiad.  than 
appropriata  aacurity  labala  muat  ba  attachad  according  to  Air  Forca 
labaling  raquiraaanta . 


GLOSSARY; 

gfgyy ^ . fyoy ^aiona .  Tha  totality  of  thraata,  vulnarabilitiaa, 
and  protaction  '’aaehaniaaa  involvad  with  datarmining  whathar 
coaputar /comaunicationa  aaaata  can  ba  coapromiaad  through  data, 
procaaa,  or  abuaa  violationa.  Sacurity  proviaiona  axiat  acroaa  tha 
adainiatrativa,  ayataa,  and  facility  catagoriaa. 


35S£2N§s-I2f5ISiiSII2£’5i 

‘‘A7iT'’fhara  ia  no  raquiraaant  for  aacurity  proviaiona  to  ba 
iaplamantad  in  aoftwara. 


RESPONSE  .RATIONALE  ; 


BSSSQUSE^SSQBMl _ - _ 

""cciipLiTSLV^ISAGRES  »  1 


CCMPLSTELV  AGRSS? 


Al>149 


QU^TTION  DATA  SHEET 


# 

OuM^lon  NuKbar  SCM(ID>  *  020 

OyESTIQN:  Th«  idantlfiestlon  raquirsRants  for  post  doploymont  support 
srs  sdsquatsly  sddrssssd  in  ths  CRLCHP  < CRISP »  0/S  CUP) . 


ACTIVITY <?) ;  Opsratlon  Support 

gyglfAW^Ijg^? ;  Th#  crisp  snd  0/S  CHP  (or  ths  JLC  CoRputsr  Rssourcss 
Lifs  Cyels  Msnsgsssnt  Plsn  -  (HtLCMP)  srs  ths  ksy  planning  docussnts 
for  opsration  support  configuration  ssnsgsssnt.  Ths  CRISP »  0/S  CHP, 

snd  (HtLCMP  srs  Intsndsd  to  bs  living  docussnts,  svolving  to  provids  a 
currant  visw  of  ths  support  svolution  of  ths  cosputsr  rssourcss 
including  configuration  nanagsssnt  fssturss.  Ths  CRISP  (first 
vsrsion)  is  rsquirsd  ssrly  in  ths  lifs  cycls,  at  Isast  prior  to  full 
seals  dsvslopssnt.  Ths  0/S  CMP  is  rsquirsd  prior  to  ths  and  of  ths 
full  acala  dsvslopssnt.  Ksy  to  adequacy  is  ths  cospatibility  of  ths 
opsration  support  configuration  idsntif ieation  and  ths  dsvslopssnt 
contractor  configuration  identification. 

GLOSSARY; 


S£SE2NSS.B3II2NA^£i 


RESPONSE  SCORE: 
”(C0MPLiTELY"5lSAGRii' 


•  1,2, 3, 4, 5, 6  >  COMPLETELY  AGREE) 


QUESTION  DATA  SHEET 


QuMtion  Numbar  SCM(ID>  -  021 

QUESTION;  Tha  autoaatad  support  tools  for  post  daploysant .  support  of 
confisuration  idantlfication  sra  adaquataly  sddrassad  in  tha  CRLCHP 
(CRISP,  0/S  CMP). 

ACTIVITY <S);  Oparstion  Support 

EXPLANATIONS;  Tha  CRISP  and  0/S  CMP  (or  tha  JLC  Cosputaf  Rasourcaa 
Lifa  Cyela  Hanagasant  Plan  -  CRL(niP}  ars  tha  kay  planning  doeusanta 
for  oparation  support  configuration  sanagasant.  Tha  CRISP,  0/S  CMP, 
and  CRLCHP  ara  intandad  to  ba  living  docusants,  avolving  to  provida  a 
currant  viaw  of  tha  configuration  sanagasant  faaturas  along  with  tha 
avolution  of  tha  systas.  Tha  usa  of  autosstad  support  tools  during 
davalopsant  and  transition  of  thosa  tools  to  usa  during  post 
daploysant  support  is  an  isportant  eonsidaration  for  tha  ovarall 
anhancasant  of  aoftwara  supportability .  Tha  lack  of  such  tools  to 
sanaga  tha  configuration  idantifieation  indax  of  tha  various 
basalinaa  should  ba  considarad  a  sarious  dafiCiancy. 


SsSPSN§S-I!f§ISySIIQl!5i 


RggPONSg  RATIgNAV? ; 


RggPONSE  SCORE;  _ 

"<COMPLETELY”d  IS  AGREE  ■  1,2, 3, 4, 5, 6  ■  COMPLETELY  AGREE) 


msmassssmKKBssmKmm 


Al-lSl 


SOFTWARE  CONFIGURATION  HANAGEMENT  CONTROL 


Tha  quaationa  SCH(CC)>001  through  SCH(CC>-023  addroas  iaauaa  of 
aoftwara  configuration  control  for  tha  procuranant,  davalopnant 
contractor*  and  oparatlon  aupport  actlvltiaa.  Configuration  control 
ia  tha  major  procaaa  of  configuration  aanaganant.  It  ia  tha  procaaa 
by  which  changa  daciaiona  ara  aada  (by  tha  Configuration  Control 
Board  atructura)  *  adainiatarad  (by  tha  Configuration  Nangaganant 
Offica  of  tha  Progran  Offica  -  or  aquivalant)*  and  iaplaaantad  (by 
changa  control  paraonnal  appropriate  to  tha  Ufa  cycle  atate  of  tha 
aoftwara) . 

Tha  daciaion-aaking  part  of  configuration  control  datarninaa 
whether  propoaad  changaa  to  a  controlled  docuaant  or  aoftwara  item 
will  be  beneficial  to  tha  Govarnaant  in  tarna  of  operational 
affect ivanaaa*  aupport  naada*  coat*  and/or  achadula.  Tha  changa 
adminiatratlon  and  iaplaaantation  parta  anaura  that  all  approved 
changaa  to  a  configuration  ara  properly  incorporated  in  tha  affected 
docuaanta  and  aoftwara  coda  and  that  no  other  changaa  find  their  way 
in. 


Configuration  control  focuaaa  on  tha  approved  baaalinaa: 

(1)  Functional  Baaalina 

(2)  Allocated  Baaalina 
O)  Product  Baaalina 


and*  tha  appropriate  davalopaant  contractor  Davalopaant  Baaalinaa. 

Software  itaaa  and  docuaanta  that  ara  not  baaalinad  ara  not 
aubjact  to  baaalina  configuration  control*  but  aay  be  placed  under 
internal  (contractor  or  aupport)  configuration  control  during  tha 
aoftwara  life  cycle.  Baaalina  configuration  control  raliaa  upon 
interaction  aaong  tha  procuraaant*  davalopaant  contractor,  and 
.  operation  aupport  activitiaa.  The  adequacy  of  the  development 
contractor  internal  configuration  control  ia  important  ainca  tha 
plana*  tachniquaa*  and  toola  would  be  beneficial  for  tranafar  to  tha 
operation  aupport  activity  for  uaa  during  tha  poat  deployment  life 
cycle  phaaa.  From  tha  operation  aupport  viewpoint*  it  ia  not 
aufficiant  that  tha  development  contractor  activity  can  control  tha 
baaalinaa  aufficiantly  to  deliver  a  configured  product.  For  amooth 
tranaition  it  ia  nacaaaary  that  tha  configuration  control  procaaa  can 
be  tranaitlonad  to  or  la  compatible  with  tha  aupport  activity 
configuration  control  procaaa. 

Interface  control  ia  alao  a  vary  important  aapact  of 
configuration  control*  aapacially  with  ayatama  which  have 
multi-aarvica  operational  raquiramanta  and  ayatama  which  require  more 
than  one  alamant  for  development  or  aupport.  Separata  control  boarda 
and  review  boarda*  and  integrated  working  groupa  ara  required  to 
manage  the  complicated  development  and  aupport  raquiramanta. 


QUESTION  DATA  SHEET 


OuMtlon  Nu»b«r  SCMCCC)  *  001 

Oy^STION:  Tha  procuramant  policy,  atandarda,  and  convantlona  appllad 
to  tha  control  of  aoftwara  configxiration  Itaaa  ara  adaquata. 


ACTIVITY <S) :  Procuraaant 

Control  of  coaputar  aoftwara  configurad  itaaa  and 
procaduraa  for  changing  configurad  itaaa  ara  daacribad  in  diraetivaa, 
ragulationa,  and  guidalinaa:  AFH  69**3,  AFR  97-4,  AFSCP  SOO-3,  OoDD 
9000.29,  APR  aOO-14,  OoOO  9010.21,  AFSCP  600-7.  It  ia  tha 
rasponalbility  of  tha  procuraaant  activity  to  aaaura  that  propar 
policy,  atandarda,  and  convantiona  ara  raquirad  for  tha  control  of 
configurad  Itaaa.  Contractor  coapllanca  docuaanta  which  could  ba 
raquirad  includa  MIL-STD-460A,  HIL-STD-4d2,  HZL-STD-463A, 
KIL-S7D-490A,  and  DoD-STO-2167. 


G^ggSARY; 

Sgf  -  Itaaa .  Softwara  alaaanta  which  ara’ 

daaignatad  for  configuration  control  by  tha  contractual  raquiraaanta . 

Cqn^i^ol .  Tha  procaaa  of  ayataaatie  ovaraight  of  changaa  to  a 
con£igurad  itaa  and  ita  functional  ralationahip  with  othar  configurad 
itaaa  for  purpoaaa  configuration  aanagaaant. 


R5SP0HS5  INSTRUCTION^ ; 


RggPQNg^  RAT;9NAVg; 


RESPONSE  SCORE; _ 

"TcOMPLETSLY'oISAGREE  -  1,2, 3, 4, 9, 6  ■  COMPLETELY  AGREE) 


Al-193 


QUESTION  DATA  SHEET 

OuMtlon  Nuab«r  SCMCCC)  -  002 


QUESTION*  Th«  procur«K*nt  •etlvit)'  has  iaplaaantad  adaiquata  softwara 
configuration  aanaganant*  baaad  upon  ragulationa»  to  control  tha 
funetiondl  and  physical  charactariatics  of  all  CSCZa. 

Procuraaant 

ATTPWS :  Tha  procuraaant  prograa  aanagar  is  raaponaibla  for 
iaplaaanting  a  configuration  aanagaaant  prograa  baaad  on  APR  65-3 
that  will  idantify*  docuaant*  and  control  tha  functional  and  physical 
charactariatics  of  all  CSCZa  undar  davalopaant.  Primary  planning 
document  is  tha  Program  Kanagamant  Plan  (PHP>.  Othar  activitiaa 
includa:  coordinating  raquiramanta  with  using  and  supporting 
aganciaa;  raviawing  contractor  plana |  auditing  contractor 
iaplaaantation  of  plana;  anauring  configuration  idantif icationa  for 
all  CSCZa  ara  proparly  docuaantad;  controlling  anginaaring  changes  to 
baaalinaa;  providing  interface  control  for  distribution  of  changes; 
preparing  PKRT  package  for  transfer  to  tha  operation  support 
activity. 

SkSSS&S!Cl 


Sa2£S^§S-IS§IBLiSIIQSSi 


’ PgSPOWSE  rationale: 


“S3PCMSS  SCC!^E! 
“<c5MPLSTELY“DISAGRii”« 


1.2. 3. 4. 5. 6  >  COMPLETELY  AGREE) 


A1-1S4 


QUESTION  DATA  SHEET 


QuMtlon  Nuiibar  SCHCCC)  -  003 

smsii&Ni  Th«  procuramant  conf Iguration  ■•nagamsnt  planning  docuaants 
contain  au^flciant  guidanea  for  configuration  control. 


ACTIVITY t  Proeurmmmnt 

BfPW^y^TIOWS;  Tha  aajor  planning  docuaanta  for  proeuranant  ara  tha 
Program  Hanagaaant  Plan  (PMP>»  tha  Raquaat  for 
Propoaal(RFP}/Stataaant  of  WorkCSOW),  tha  Contract  Data  Raquiraaanta 
Liat<CDRL}«  and  tha  Coaputar  Raaoureaa  Intagratad  Support  Plan 
(CRISP) .  Tha  Joint  Logiatica  Coaaandara  aoftwara  atandardization 
program  haa  a  raplacamant  for  tha  CRISP  callad  tha  Coaputar 
Raaoureaa  Lifa  Cycla  Hanagaaant  Plan  (CRLCMP) .  AFR  AOO-IA  calla  for 
tha  ineluaion  of  configuration  managamant  eoncapta  in  tha  PHP 
including  configuration  control  (apacifieatlon  and  intarfacaa) .  Tha 
RFP/SOW  dafinaa  tha  a  amaet  aeopa  of  tha  davalopmant  contractor 'a 
configuration  control  raaponaibilitiaa.  Tha  CDRL  idantiflaa  all 
dalivarabla  data  itama  including  CSCIa  which  tha  davalopmant 
contractor  auat  dalivar  and  control.  Tha  CRISP  muat  to  includa 
aaaignmant  of  configuration  control  raaponaibilitiaa  during  poat 
daploymant  aupport  with  datailad  proeaduraa  dafinad  in  tha 
Oparational/Support  Configuration  Managamant  Proeaduraa  (0/S  CMP). 

iksssmi 


RESPONSE  INSTRUCTIONS: 


RESPONSE  RATI0W/\L^: 


SiSES-£§g_SSQSii _ 

TcOHPLETELY'BiSAGREE  >  1.2,3«4,9,6  >  COMPLETELY  AGREE) 


QUESTION  DATA  SHEET 


QuMtlon  NuBbttr  8CH<CC)  -  004 

flUESXIQiil  davvlopnsnt  contractor  configuration  aanagaaant 

activltlaa  ara  adaquataly  aonltorad  by  tha  procuraaant  activity. 


ACTIVITY <S) ;  Procuraaant 

g^Vi\NATI0NS;  Tha  procuraaant  activity  can  monitor  davalopaant 
contractor  configuration  aanagaaant  activltlaa  through  contractor 
doeuaanta  and  raporta#  othar  program  offica  araaa  (a.g.»  quality 
aaauranca)*  configxiration  audita »  and  avaluation  chackliata  (a.g.. 
FCA  and  PCA  praparation  chackliata  in  MIL-STD-1S21B»  ECP  praparation 
chackliata  in  0oD-8T0-4d0A  and  aodifiad  by  MIL-STD-4d3A,  Coaputar 
Raaourca  Managar'a  Chackliat  baaad  on  AFR  600- 14 »  and  attachaanta 
3«4»5  of  AFSCP  600-7  on  RFPa  and  contracta). 


&itfiSSARYl 


5£5E2HSi-ISi5ISSSII2!!Si 


RESPONSE  RATIONAL^! 


’7c0MPLiTELY”DISAGRii”»”l.2,3.4.5,6  ■  COMPLETELY  AGREE) 


ibh  mi  mm  urn  ■■  w  > 


rvwuvuB  ufivvi 


QUESTION  DATA  SHEET 

QuMtlon  Number  SCM(CC>  -  009 

QUESTION:  Th«  proeur«K«nt  configuration  control  procaduraa  for  tha 
Claaa  I  and  Claaa  II  changaa  (or  aquivalant  catagoriaa>  ara  adaquata. 


ACTIVITY <9 LI  Procuraaant 

Claaa  I  changaa  involve  primarily  any  major  changaa 
to  tha  Configuration  Baaalinaa*  contractual  proviaiona*  aupport 
compatibility,  and  ao  forth.  Claaa  II  changaa  ara  for  minor 
daficianciaa  and  do  not  ganarally  raquira  procuraaant  approval,  but 
thara  ahould  atill  axiat  a  machaniam  for  procuraaant  raviaw  alnca 
many  timaa  Claaa  II  changaa  could  cauaa  aida  affacta  which  might 
raault  in  tha  change  being  raclaaalfiad  aa  a  Claaa  I  change.  A  Claaa 
II  change  la  juatifiad  if  it  banafita  tha  development  contractor  and 
ia  not  datrinantal  to  tha  Govarnmant  (proeuramant  and  operation 
aupport).  Guidance  for  Claaa  I  and  Claaa  II  changaa  la  found  in 
DoD-STD^AACA,  NIL-STO-AdSA,  and  other  govarnmant  internal  and 
compliance  documanta. 

Sk2SSdSVi 

Engineering  change  which  affacta  a  Baaalina 
Identification,  performance  outaida  atatad  tolarancaa,  external 
interface  character iatica,  budgat/raaourca  raquiramanta,  or  other 
factora  of  major  aigniflcanca  to  tha  operational  effectiveneaa  or 
suitability  of  the  aoftware  product. 

j  Hnginaaring  change  not  elaaaified  aa  Claaa  I. 
Ir.cludea  minor  changaa  auch  aa  typographical  errora  in  documanta, 
addition  of  commenta  to  aource  code,  changaa  to  adaptation  data  auch 
aa  data  baae  parametera,  and  aingla  racompilationa. 

-:N3T;^ucT;gN§  t 


RATION^yg; 


se3RF! 


Al-157 


■  wvimiitew 


anat  uvmmiunai  ln^nnnt^^gKMn^l^B^wtapl^>wafmi^^■u^nwnl1n^^iw^re^^v»wt^mnnl^q^lanl■wu»1 


I 


QUESTION  DATA  SHEET 


QuMtlon  Numbar  SCIKCC)  ~  006 


QUESTION;  7h«  us«  of  doviotion*  and  woivors  by  tho  dovolopmont 
contractor  which  could  affact  tha  aupportabllity  of  tha  aoftwara  haa 
baan  adaquataly  controllad  by  tha  procuramant  activity. 


tSimiTllSll  Procuramant 


daviationa  and  waivara  must  ba  carafully 
monitorad  for  tha  poaaibla  advaraa  affact  upon  aoftwara 'a 
aupportability  avan  though  tha  oparational  affactivanaaa  may  not  aaam 
to  ba  diractly  affactad.  Aa  an  axampla,  if  tha  uaa  of  a  High  Ordar 
languaga  ia  waivad,  than  tha  aupportability  of  tha  aoftwara  haa  baan 
affactad . 


SLOSSARY; 


:?E5?CNSE  instructions; 


RESPONSE  rationale: 


Al-158 


•J’ 


QUESTION  DATA  SHEET 

Question  Number  SCH(CC>‘  *  007 

iiStZZQitl  ?**ocurement  beeellne  control  forme  are  adequate. 


A  Procurement 

Si^SL^U^IISlfSi  procurement  baseline  control  forms  might  include: 
Engineering  Change  Proposal  <ECP>#  Specification  Change  Notice<SCN)» 
I^equest  for  Waiver,  Software  Deficiency  Report,  and  Software  Change 
Report . 


© 


S^OSSARY: 


QUESTION  DATA  SHEET 


QuMtlon  NuBb«r  SCMCCC)  -  OOd 

QUESTION ;  Thtt  procurvmsnt  configuration  control  board  procaduraa  ara 
adaquata. 


A?T?^SIYiS2l  Procuraaant 

Tha  procuraaant  CCB  procaduraa  include  conduct  of 
maatinga,  maintananca  of  raeorda^  control  of  tha  baaalinaa, 
integration  of  hardware  and  aoftwara  concarna  during  tha  change 
procaaa,  formation  of  a  aaparata  aoftwara  configuration  control  board 
if  tha  complexity  of  the  aoftware  ao  juatifiea  auch  a  board,  and 
control  of  interoperability  interface  probleaa  acroaa  any  aaaociated 
ayatema.  Change  control  procedurea  ahould  provide  for  careful 
evaluation  of  all  SCPa  according  to  conaideratlona  aentionad  in  AFP 
SS<-3  and  AFSC?  dCiO-7.  In  particular,  CSCZ  changea  which  have  an 
effect  on  multiple-  location  applicationa,  nuclear  aafaty,  aedurity, 
coat,  achedule,  other  CSCIa,  other  hardware  or  interfacea,  and 
aupport  reaoiircea  muat  be  carefully  analysed  for  overall  benefit  to 
the  Sovernment. 


PiSPCNSE  .NATIONALS : 


QUESTION  DATA  SHEET 


QuMtlon  Number  SCHCCC)  -  009 

QUSSTIoy :  The  procurement  proceduree  for  turnover  end  trenefer  of 
configuretlon  control  to  the  operetion  support  ectlvity  hee  been 
edequetely  plenned. 

ACTIVITY <S);  Procurement 

The  AFR  AOO-IS  governs  system  turnover »  end  the  CRLCMP 
(CRISP  end  o7s  CHP)  contsins  specific  guldsnce  ss  to  the  form  end 
format  of  whet  the  turnover  et  PHRT  will  be.  Key  to  the  edequscy  of 
this  process  is  the  amount  of  early  planning  and  the  specificity  of 
the  details  in  the  CRLCMP  (CRISP  end  0/S  CMP)  et  the  milestone  (end 
interim  milestone)  decision  points.  Frequently  the  mare  existence  of 
such  documents  does  not  imply  that  they  ere  at  ell  sdaquate. 


GLOSSARY : 


RSSPONSI  INSTRUCTIONS: 


SiSPONSg.RAT^SNALgi 


RggPONSg  SCORE:  _ 

"(c5mPLSTELY"dIsaGREE  ■  1.2,3,4.S,6 


COMPLETELY  AGREE) 


QUESTION  DATA  SHEET 

QuMtion  Nu»b«r  SCIKCO  *  010 


oy^STIOM^  D«v«lopii«nt  contractor  configuration  control  atandarda  and 
eonvantiona  can  ba  tranaitio'nad  to  operation  aupport  atandarda  and 
convent Iona . 


ACTIVITY  T  Development  Contractor 

Th*  AFR  aoo-is  governa  ayatea  turnover*  and  the  (CRISP 
and  0/S  CHP>  containa  apecific  guidance  aa  to  the  fora  and  format  of 
what  the  turnover  at  PHRT  will  be.  Key  to  the  adequacy  of  thia 
proceaa  ia  the  amount  of  early  planning  and  the  apecificity  of  the 
detaila  in  the  CRISP  and  0/S  CHP  at  the  mileatona  (and  intarim 
mileatone)  deeiaion  pointa.  Frequently  the  mere  exiatance  of  auch 
docuaanta  doea  not  imply  that  they  are  at  all  adequate. 


9V0SSARY; 


RS5P0M§E.IN5TRyCTigM52 


RESPONSE  RATIONALE: 


RESPgNSE_SCgRE2 
““cCMPLlTiilv“5*2ACsii”«~l  ,2.3 


9  •  CCKPLETELY  AGPSZ: 


Al-162 


QUESTION  DATA  SHEET 


QuMtlon  Nunbar  SCHCCC)  -  Oil 

dUSSIICtil  davalopmant  contractor  configuration  control  board  haa 
an  adaquata  intarfaca  with  tha  proeuraaant  activity  configuration 
control  board. 

tSimiUSSll  Oavalopaant  Contractor 

'  AFR  S00-14  raquiraa  coaputar  program  configuration 
aanaganant  to  ba  intagratad  into  tha  ovarall  ayatam  configuration 
aanagaaant  and  acroaa  all  cognizant  organization  alaaanta.  Zntarfacaa 
ara  vary  important*  and  ona  of  tha  moat  important  ia  tha 
communication  batwaan  tha  procuramant  and  davalopaant  contractor 
configuration  control  boarda.  Tha  CCBa  ara  tha  official  organizationa 
ampowarad  to  act  on  all  propoaad  changaa.  The  primary  changaa  which 
would  raquira  intarfacing  ara  Claaa  Z  changaa.  MIL*STD-483A  ia  tha 
primary  davalopmant  contractor  complianca  ragulation. 


SiS55ARYi 


"Tc5?.?Li7iL7“3 :3AGRii”»”i ,  2 .  a .  4 .  s .  s 


COMPLETELY  AGSSE) 


Al-163 


QUESTION  DATA  SHEET 


QuMtlon  NuBb«r  SCHCCC)  -  Ol2 

QUESTIOSl  d«v«lopR*nt  contractor  configuration  control  board 
procaduraa  ara  adaquaba  to  diatinguiah  batwa«n  hardwara  and  aoftwara 
failuraa. 

Davalopaant  Contractor 

gypif/^K/^TIONS :  For  larga  ay  at  ana  aaparata  hardwara  and  aoftwara  boar  da 
may  ba  aatabliahad  undar  a  ayataa  laval  board.  Failura  raporting  auat 
adaquataly  charactariza  failuraa  ao  dataraination  of  tha  aourca  of 
tha  failura  ia  poaaibla.  Such  raporta  and  aolutiona  to  failuraa  can 
than  ba  procaaaad  nor a  adaquataly  by  tha  'Control  boarda. 


GLOSSARY! 


RgSPONSS^INSTRUCTigNS^ 


RE^PONSE^RATIONALE! 


S£S£SNS£.ScgR£i 


,3, ^.2. 5  «  3C:!?L3T21?  AGREE) 


Al-164 


QUESTION  DATA  SHEET 


OuMtlon  Number  SCH(CC)  -  013 

QUSSIISNl  The  developeent  eontrector  con£iguratlon  control  procodurox 
cen  be  treneltioned  to  or  ere  competlble  with  the  operation  support 
ectlvlty  planned  configuration  control  procedures. 

*  Development  Contrector 

BffLANATIONS;  The  CPLCHP  (CRISP  and  0/S  CMP>  deecribea  the  operation 
support  planned  configuration  control  procedures.  usually  in 
accordance  with  APR  57-4.  Contractor  compliance  documents  include 
DoO-STD‘‘4dOA  end  MIL-STD-4d3A.  The  contractor's  configuration  control 
procedures  should  be  documented  in  a  Software  Configuration 
Management  Plan. 


glossary; 

e 


RSSP0N?E,3;W§TRUCTIC>_rg! 


response  „RATI0W.V,g! 
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QUESTION  DATA  SHEET 


QuMtlon  Nuaib«r  SCN(CC)  -  014 

aaSSIISai  Th«  d®v*l  3pT.4nt  contractor  autesiatod  support  'tools  for 
configuration  control  of  baaalinaa  and  intarnal  davalopnar.t 
Idantlficationa  la  adaquata. 

ftyriVIXY<?>  *  Oavalop.T.ant  Contractor 

Tha  autoaatad  support  tools  might  includa  a  control 
library#  automated  procaduras  to  lock  out  multipla  paraonnal  from 
modifying  a  aodula  at  tha  aama  tima#  automatad  varaion  and  variation 
idantlf ication*  automatad  tracaability  of  raquiramanta .  daaign.  coda# 
and  taat  alamanta#  and  so  forth.  Tha  uaa  of  automatad  tools  is 
aaaantial  for  comp  la::  softwara  systaas. 


NSSPONSS 


RESPONSE  RATICNALE; 


5S5Pe?iSE_sccPEi 


CCilPtETEL'.'  lISnGP.EE  »  L^Z.Z 


STION  DATA  SHEET 


QuMtion  Number  SCH<CC>  -  OIS 

QUgyr^QN:  Th*  davalopacnt  contractor  software  change  control  for ns 

are  adequate. 

/^yriVITYC?) :  Davalopaant  Contractor 

EffiVASAXISS?  T,  change  control  foraa  should  seat  raqulraaants  of 
applicable  Govarnaant  atandards*  and  provide  sequential 
identification  suitable  for  logging#  filing#  rafarenca#  and 
retrieval.  Content  should  adequately  addraaa  source  of  change 
request#  reason  for  request#  type  (anhancaaant#  correction)  of 
request#  effect  of  change  on  the  ayatee#  resource  requireeenta  to 
inpleaent  the  change#  and  adeinietrative  information  'Such  as  approval 
signatures  required  and  expected  (actual)  change  milestone  dates. 

CtOggARYi 


RESPONSE  IHSTRUCTIOWS:  . 


RESPONSE  RATIONALE; 


RESPONSE  SCORE: 

“(COMPLiTiLY'SiSAOiii  •”l,2,3,4#5#6  ■  COMPLETELY  AGREE) 


QUESTION  DATA  SHEET 


QuMtlon  Nuabar  SCH<CC) 


Subcontractor  configuration  Itam  control  practicaa  ara 
monitorad  by  tha  davalopmant  contractor. 

ACTIVITY <S) ;  Davalopmant  Contractor 

S3$SL^2!i^XIS2fSi  thara  la  a  aubcontr actor »  It  will  ba  nacaaaary  that 
tha  davalopmant  contractor  raqulra  configuration  control  practicaa 
aiallar  to  thoaa  raqulrad  by  tha  procuramant  activity.  If  thla  la  not 
dona*  than  tha  davalopmant  contractor  will  ba  raqulrad  to  ratroflt 
tha  control  practicaa  of  tha  aubcontraetor .  Tha  control  practicaa  of 
tha  aubcontraetor  muat  ba  carafully  monltorad  to  aaaura  compatibility 
and  propar  Intarfacaa  of  tha  cognizant  configuration  control  boarda. 


-JLCSSA.^Y: 


A/'3:  No  auboontractora  hava  raaponalblllty  for  davalopmant  of 
aoftwara  products 


SZSPONSS  SATICNALS: 


QUESTION  DATA  SHEET 

Quastion  Nunbar  SCNCCC)  -  017 

QUESTION:  Configurad  Itama  which  iaplamant  aafaty  provlaiona  ara 
adaquataly  controllad. 

Davalopmant  Contractor 

Configurad  itaaa  which  iaplamant  aafaty  proviaiona  ara 
fraquantly  controllad  by  aoftwara.  Thia  aoftwara  auat  ba  ^  adaquataly 
idantifiad  aa  aff acting  aafaty.  Safaty  proviaiona  ara  cloaaly  ralatad 
to  tha  raliability  of  aiaaion  critical  coaponanta,  aafaty  of  niaaion 
paraonnal*  nuclaar  aff acta*  and  ao  forth. 


i  a 


A/s:  Thara  ia  no  raquiraaant  for  aafaty  proviaiona  to  ba 
iaplamantad  or  controllad  by  aoftwara 


RESPONSE  RATIONALE: 


RESPONSE  SCORE: 
'cCONPLETiLY’DISAGRii' 


1*2*3*4*5*6  >  COMPLETELY  AGREE) 


Al-169 


QUESTION  DATA  SHEET 


Qu««tlon  Nunbar  SCN<CC>  -  Old 


QUESTION ;  Software 
computer /comRunleatlona 
eontrollmd. 


configured 

security 


itans 

proviaione 


which 


ara 


iaplanant 

adaquataly 


Davalopmant  Contractor 


Software  which  implaaanta  computar/communlcatlon 
security  is  particularly  important.  Any  such  softwsrs  items  must  ba 
adequately  controlled  as  part  of  the  trusted  computer  base.  If  the 
configured  software  item<a>  are  thaaaalvea  classified.  than 
appropriate  security  labels  must  be  attached  according  to  Air  Force 
labeling  requiranants  and  access  control  of  such  items  must  ba 
enforced. 


GLOSSAHY^ 

^  The  totality  of  threats.  vulnerabilitiaa . 
and  protection  aachanisms  involved  with  determining  whether 
computer /communications  ssssts  can  ba  compromised  through  data, 
process,  or  abuse  violations.  Security  provisions  exist  serosa  the 
administrative,  ayatem.  end  facility  categories. 


3§§292i§I-I2!§I3y£II22§I 

A/6:  There  ia  no  requirement  for  aeeurity  provisions  to  ba 
implemented  in  software. 


R^PQNSE  RATIONALE; 


3£SSSI!Si.S90S£l 


# 


Al-170 


I 


QUESTION  DATA  SHEET 


QuMtion  Nu»b«r  SCHCCO  -  019 

SUSSIlQlii  Di«t  rlbution  of  conflgurod  Itoai  ehangoa  from  thm  opormtlon 
support  mctlvlty  to  thm  flsld  1«  mdsqumtmly  controllmd. 


ACTIVITY<S) ;  Dsvmlopamrt  Contractor 

gj^^AN^ITIQ^^ ;  Tha  diatribution  must  aatiafy  applicabla  standards  and 
r^ulstlons  for  tachnlcal  ordsra  as  wall  aa  ths  mission  critical 
Isauss  of  coi'ractnaaa  of  changes  and  timsllnsss  of  changaa. 
Xntsrfacaa  among  oparatlon  support  and  fiald  support  organization 
alamanta*  Including  configuration  boarda  and  logistics  supply  for 
tachnlcal  orders ,  ara  critical  to  tha  auccasa  of  tha  distribution 
process. 


CtSSSASYi 


5=§£21{SE_INSTRUCTigNSi 

Tisialinasa  of  tha  distribution  process  after  anginaarlng  ralaasa 
is  complete  Is  one  of  tha  critical  Issues  to  consider.  Although  there 
are  no  fixed  standards,  it  seams  reasonable  that  no  more  than  90z  of 
tha  time  spent  for  engineering  should  be  required  to  complata  the 
distribution  to  the  fiald.  This  "parcantaga'*  Is  bound  by  a  lower 
absolute  value  of  time  required  based  upon  physical  limitations 
Ce.g..  prom  burning,  tachnlcal  order  genoration>  of  the  distribution 
process. 

RESPONSE  RATIONALE: 


RESPONSE  SCORE: 


Al-171 


QUESTION  DATA  SHEET 


QuMtlon  NuBib«r  SCM(CC)  **  020 

cussxi&lil  Th«  configuration  control  rosponaibility  for  integrating 
computer  resources  into  the  system  has  remeined  centrslizad 
throughout  the  life  of  the  system. 

All 

g?^LANATI0WSl  Although  orgenizetionel  elements  (e.g.,  HQ~TAC» 
HQ-AFLcT  may  have  configuration  control  responsibilities  for  separata 
elements  of  the  system  (e.g.#  software*  hardware)  there  should  be  a 
centralized  control  point  for  ell  decisions  (perhaps  s  set  of 
configuration  control  boards) .  As  the  software  is  passed  from  the 
development  contractor  to  the  operation  aupport  activity  the 
configuration  control  responsibilities  are  passed  from  the 
centralized  development  configuration  control  to  the  central izaed 
aupport  configuration  control*  with  appropriate  planning  and 
attention  to  the  actual  transfer  of  responsibility. 

SLQSSARY! 


A 


RESPONSE  INSTRUCTIONS; 


QUESTION  DATA  SHEET 


Question  Numb«r  SON  (CO  -  021 

QUESTION;,  The  conf iguration  control  requirements  for  poet  deployment 
support  ere  adequately  addressed  In  the  CRLCMP  (CRISP  end  0/S  CNP) . 


Operation  Support 

The  CRISP  and  0/S  CNP  (or  the  JLC  Computer  Resources 
Life  Cycle  Kanegement  Plan  ~  CRLCNP>  ere  the  key  planning  documents 
for  operation  support  configuration  management.  The  CRISP,  0/S  CMP, 
and  CRLCMP  are  intended  to  be  living  documents,  evolving  to  provide  a 
current  view  of  the  configuration  management  features  along  with  the 
evolution  of  the  systan.  The  CRISP  (first  version)  is  required  early 
in  the  life  cycle,  at  least  prior  to  full  scale  development.  The  0/S 
CMP  is  required  prior  to  the  end  of  the  full  scale  development.  Key 
to  adequacy  is  the  compatibility  of  the  operation  support 
conf igurstion  control  and  the  development  contractor  configuration 
control  procedures  and  automated  tool  support. 

2iCS5ARYl 


•  MW  ^  ^  W  ^4^  «  « 


RESPONSE  RATIONALE ; 


RSEPCMEE  3C2RE: 


*  -  I— i  .  iw ;  .nGnii  - 


Al-173 


QUESTION  DATA  SHEET 


# 

QuMtlon  Nuab«r  SCM(CC>  -  022 

SliSSIISHl  operation  support  configuration  control  boards  ara 
adaquataly  dafinad  to  handle  software  changes. 

^9TIVITY<S) :  Operation  Support 

g^g^^NATypNS;  The  CRISP  and  0/S  CMP  (or  the  JLC  Coaputar  Raaourcaa 
Life  Cycle  Hanagamant  Plan  *  CRLCHP)  ara  the  key  planning  docuaanta 
for  operation  support  configuration  aanagaaant.  The  CRISP »  O/S  CHP* 
and  CRLCHP  ara  intended  to  be  living  docuaanta*  avolving  to  provide  a 
currant  view  of  the  configuration  aanagaaant  features  along  with  the 
evolution  of  the  ayataa.  The  configuration  control  boards  along  with 
specific  board  raaponaibilitiaa  should  be  dafinad  in  the  CRISP  and 
the  0/S  CHP.  It  is  not  enough  to  indicate  that  a  given  directive* 
regulation*  standard*  or  guideline  will  be  followed.  Specific  detail 
as  to  the  board  function*  relationship  to  the  organizaitonal 
structure*  interface  reaponaibilitiea*  and  so  forth  should  be 
included  in  the  operation  support  configuration  aanagement  plan  and 
procadurea . 

ssrSsmjL 


S23PCNSS  INSTRUCTIONS; 


RESPONSE  RATIONAL.; ; 


2r522^§g_scgR£i. 

,.1.3.4.3,5  »  CCNPLSTEL?  AGREE> 


Al-174 


QUESTION  DATA  SHEET 


QuMtion  Number  SCMCCO  -  023 

QUESTION ;  Th*  automatad  aupport  toola  for  poat  daployaant  aupport  of 
configuration  control  ara  adaquataly  addraaaad  in  tha  CRLCHP  (CRISP 
and  0/S  CMP). 

ASIiyilXiSH  Operation  Support 

gypyfttf^TIONS ;  Tha  CRISP  and  0/S  CHP  (or  tha  JLC  Conputar  Raaourcaa 
Life  Cycle  Hanagaaant  Plan  *  CRLCHP)  ara  the  key  planning  docuaanta 
for  operation  aupport  configuration  aanageaant.  The  CRISP*  0/S  CHP* 
and  CRLCHP  are  intended  to  be  living  docuaanta*  avolving  to  provida  a 
currant  viaw  of  tha  configuration  managaaant  faaturaa  along  with  tha 
evolution  of  the  ayatea.  The  uae  of  automated  aupport  toola  during 
davalopaant  and  tranaition  of  thoae  toola  to  uaa  during  poat 
daploynant  aupport  ia  an  important  conaidaration  for  tha  overall 
anhancamant  of  aoftwara  aupportability.  Tha  lack  of  auch  toola  to 
aaaiat  in  tha  configuration  control  of  the  varioua  baaalinaa  ahould 
ba  eonaidarad  a  aarioua  dafieiancy. 

GLOSSARY ; 


BsSPONSS^INSTRUCT’SNSi 


5S5S9r*^S-5ATigNALEi 


PSSPOJfSS  SCORE:  _  (COMPLETELY  DISAGREE  ■  1*2*3*4.5*6 

COMPLETELY  AGREE) 


A1-17S 


SOFTWARE  CONFIGURATION  KANAGEHENT  STATUS  ACCOUNTING 


Tha  quaationa  SCH(SA>-001  through  SCM(SA)-016  addraaa  Iscuas  of 
•oftwaro  configuration  atatua  accounting  for  tha  procuroaant* 
davolopaant  contractor,  and  oporation  support  act.ivitiaa. 
Configuration  status  accounting  is  ths  procsas  of  ksaping  •  track  of 
bha  configuration  idantif Ication  and  its  changaa,  and  raporting  this 
information  to  managamant.  Two  typaa  of  docuaanta  ara  producad  by 
configuration  atatua  accounting: 

(1)  Software  Configuration  Indax:  dafinas  tha  currant  approvad 
configuration  of  an  itam  in  tarma  of  its  alamanta  or 
idantif  ication  docuaanta  and  its  approvad  changaa 
<2)  Change  Status  Reports:  fov  daficiancy  and  modification 
changaa  to  a  configured  item. 

Thaaa  configuration  atatua  accounting  documents  provide  all 
activitiaa  with  visibility  and  traceability  of  baaalina 
configurations  and  their  changaa.  Coordination  of  activitiaa  and 
daciaiona  regarding  these  activitiaa  much  as  scheduled  raviawa, 
audita,  taata,  use  of  teat  raaourcaa,  raquiramanta  for  budget 
ad^uataanta  to  tha  contract,  and  so  fort)!  ara  baaed  upon 
configuration  atatua  accounting  information. 

V 


Al>176 


QuMtion  Nu«b«r  SCHCSA) 


001 


QUESTION »  Thm  procurMcnt  policy^  standArda*  and  eonvantiona  appliad 
to  tha  configuration  atatua  accounting  of  aoftwara  configuration 
itaaa  ara  adaquata. 

;  Procuranant 

Docuaantation  for  daacribing  and  raporting  tha  atatua 
of  coaputar  aoftwara  configurad  itaaa  ia  daaeribad  in  diractivaa# 
ragulationa*  and  guidalinaa:  AFR  6S-3»  AFR  97-4  >  AFSCP  dOO-3#  DoOO 
5000. 29<.  AFR  800-14,  DoOO  9010.21,  AFSCP  *  800-7.  It  ia  tha 
raaponaibility  of  tha  procuraaant  activity  to  aaaura  that  propar 
policy,  atandarda,  and  eonvantiona  ara  raquirad  for  tha  configuration 
atatua  .  accounting  of  configured  itaaa.  Contractor  coaplianca 
docuaanta  which  could  ba  raquirad  include  HIL-8T0-4d0A,  XIL-STD-4d2, 
SIL-STD-4a3A,  RIL-S70-430A,  and  DoD-STD-2167. 

GLOSSARY; 

_ _ |tama.  Software  alaaanta  which  ara 

daaignatad  for  configuration  atatua  accounting  by  tha  contractual 
raquiraaanta . 

g9n^jtgygatj^on_gfef^ua^Acsgyn^4»igt  Tha  aaana  through  which  aetiona 
affecting  CSCIa  are  record^  and  reported  to  prograa  and  functional 
aanagara. 


32S?2NS|_iNg73UCTigNSj, 


?S3?CM32_S.CgRS2 _ 

"IcCXPLiTiLV' DISAGREE  ■  1,2, 3, 4. 5, 6  ■  COMPLETELY  AGREE) 


I 

I 

I 
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QUESTION  DATA  SHEET 


# 

QuMtlon  Number  SCMCSA)  -  002 

QUESTION;  The  procurement  activity  has  implemented  adequate  software 
conf Ifuration  status  accounting#  based  upon  regulations#  to  report 
the  functional  and  physical  characteristics  of  all  CSCIs. 

tTY  V  Procurement 

procurement  program  manager  is  responsible  for 
implementing  a  configuration  managesent  program  based  on  AFR  65-3 
that  will  identify#  document#  and  control  the  functional  and  phyeical 
characteristics  of  all  CSCIs  under  development.  Primary  planning 
document  is  the  Program  Management  Plan  (PHP> .  Other  activities 
include:  coordinating  requirements  with  using  and  supporting 
agencies;  reviewing  contractor  plans;  auditing  contractor 
i.-nplementation  of  plans;  ensuring  configuration  identifications  for 
all  CSCIs  are  properly  documented;  controlling  of  engineering 
changes  to  baselines;  controlling  distribution  of  changes;  preparing 
?MRT  package  for  transfer  to  the  operation  support  activity. 

SiSSSARY^ 


INSTRUCTIONS : 


■  g  atjqn  A^f  g-; 


Al-178 


I 


QUESTION  DATA  SHEET 

QuMtlon  Number  SCH(SAJ  -  003 

procur«m«nt  configuration  managomont  planning  documanta 
contain  auffieiant  guidanca  for  configuration  atatua  accounting. 

ACTIVITY <S)  ;  Procuraiiant 

aajor  planning  documanta  for  procuramant  ara  tha 
Program  Nanagamant  Plan  (PHP)#  tha  Raquaat  for 

Propoaal  (Rr  ?) /Stataaant  of  Work  (SOW)*  tha  Contract  Data  Raquiramanta 
Liat(CORL)f  and  tha  Computar  Raaourcaa  Zntagratad  Support  Plan 
(CRISP).  Tha  Joint  Logiatica  Commandara  aoftwara  atandardization 
program  haa  a  raplacanant  for  tha  CRISP  callad  tha  Computar 
Raaourcaa  Lifa  Cycla  Kanagamant  Plan  (CRL.CHP) .  APR  800-14  calla  for 
tha  Incluaion  of  configuration  managamant  concapta  in  tha  ??!? 
(including  apaclf ication  and  intarfacas) .  Tha  RPP/SOW  dafinaa  tha 
ax-sct  acopa  of  tha  davalopmant  contractor 'a  conf  iguration  atstua 
tcccur.ting  raaponaibil  <  tiaa.  Tha  CDRL  id^ntifiaa  all  dalivarabla  data 
itama  including  CSCIa  whici^  tha  davalopmant  contractor  must  dalivar 
and  control.  Tha  CRISP  ia  to  Includa  CSCI  configuration  basalina 
rapcrting  procaduraa  to  account  for  tha  implamantation  of  chtngaa. 
Oatailad  procaduraa  ahould  ba  dafinad  in  tha  Oparational/Support 
Conf  iguration  Hanagamant  Procaduraa  (0/S  CHP) . 

I-LCSSARY; 


RESPONSE  RATIONALE: 


>S 


^  ^  f 


QUESTION  DATA  SHEET 


QuMtlon  Nuab«r  SC!{(SA>  *  CC4 

Th€  procuramiint  activity  configuration  atatua  accounting 
procaduraa  ara  adaquata. 


ACTIVITY (S).l  Procuramant 

S^ExAsi^IISSSl  procuramant  activity  configuration  atatua 

accounting  procaduraa  includa  procaduraa  to  raport  baaalina 

conf iguration  idantif ication  and  changa  atatua,  contract  managamant 
nodif icationa,  apacif ication  atatua  and  changaa,  SC?a  and  SCNa,  and 
any  othar  docunanta  which  racord  tha  aoftwara  hiatory  of  davalopmant, 
and  aupport.  This  hiatory  bacomaa  part  of  tha  PKRT  packaga  to  ba 
transfarrad  to  tha  oparation  support  activity.  Thia  hiatory  providaa 
tracoAbility  to  tha  configuration  managamant  procasa  and  tha 
rae..l'ting  aoftwara  products.  Usa  of  automated  aupport  tools  should 
tha  affactivanaaa  and  efficiency  of  tha  configuration  status 
'^ucour.ting  procaduraa. 

r..0S3AHV : 


QuMtion  Nunb«r  SCM(SA)  *  005 


d«v«lopn«nt  contractor  intornal  configuration  status 
accounting  procaduras  ara  adaquata. 

ACTIVITY <S> ;  Davalopnant  Contractor 

*  Procaduraa  should  ba  docusantad  in  a  raquirad  softwara 
configuration  managamant  plan.  Procaduras  includa:  how  information  on 
status  is  tp  ba  collactad,  varifiad»  storad»  procaaaad,  and  raportad; 
idantif ication  of  tha  pariodic  raports  to  ba  providad  and  thair 
distribution.  Information  to  ba  maintainad  ineludas:  status  of 
£picif icationa  and  proposad  changas;  raporta  of  approvad  changaa; 
;:tstu3  of  product  varsions  or  ravisiona;  raports  of  tha 
inplamantation  of  in-stallad  updatas  or  ralaaaaa;  status  of 
procuranant  suppliad  proparty. 

m  mm  «  4 
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QUSSTICN  DATA  SHEET 


OuMtlon  Numbar  SCM(SA>  -  006 

>;v;S?IC>-  *■  0«v<tlop.'R«r.t  contractor  configuration  atatua  accounting 
atandar<la  and  convantlona  can  ba  tranaltlonad  to  operation  aupport 
atandardo  and  convantlona. 

tSlDiZTlSSll  Davalopaant  Contractor 

g^^^^AT  JQ^S :  In  order  for  coaputar  raaourcaa  to  ba  aaoothly 
tranaltlonad  from  the  davalopaant  contractor  to  the  operation  aupport 
activity,  the  configuration  identification  atandarda  and  convantlona 
auat  ba  coapatlbla.  Aa  more  automated  toola  are  uaad,  thla 
raqulraaant  for  compatibility  will  ba  avan  atrongar.  Evidence  of  the 
atandarda  and  tranaltlon  atrata^y  ahould  ba  In  the  CRLCMP  (CRISP  and 
0/S  CMP)  aa  wall  aa  the  davalopaant  contractor  aoftwara  configuration 
managamant  plan. 


GLOSSARY ; 


Rg5PgM5i_iN5iRUS!iON§i 


RESPONSE  RATIONALE: 


sccrs: _ ; _  •  ® 

<':0MPLiTSLY'’DiSAGRii~-”l.2,3.4,5,S  ■  COMPLETELY  AGREE). 
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JKTT  UTi  1U\ KTS WJT  wn  ' 


question  data  sheet 


QuMtion  NuKbttr  SCH<SA)  -  007 


qygyyiON;  Th*  dttv«lopm«nt  contractor  configuration  atatua  accounting 
hmm  an  adaquata  intarfaca  with  tha  procuraaant  activity  configuration 
atatua  accounting. 

ASI2ViniS2l  Davalopaant  Contractor 

6yPWAWAT?Qt?5l  AFR  A00-14  raqulraa  coaputar  program  configuration 
managanant  to  ba  intagratad  into  tha  oyarall  ayatam  configuration 
managanant  and  acroaa  all  cognizant  organization  alamanta.  Tha  atatua 
accounting  Intarfaca  batwaan  procuraaant  and  davalopnant  contrator  ia 
tha  baaia  for  raporting  all  aignif leant  baaalina  product  actiona  and 
tha  currant  atata  of  thoaa  actiona.  Early  raaolution  of  problaaa  ia  a 
diraet  function  of  how  accurately#  conciaaly#  and  afficiantly  auch 
atatua  accounting  information  ia  praaantad.  Tha  primary  changaa  which 
raqulra  intarfaca  atatua  raporta  arm  Claaa  I  changaa.  MIL~STD-4d3A  ia 
tha  primary  davalopmant  contractor  complianea  regulation. 


CUqSSARY; 


RESPONSE .INSTRUCTIONS; 


RESPONSE  RATIONALE; 


”7c5?!?Li7ELY"5lSAGRii“”l.2.3.4.5.6  ■  COMPLETELY  AGREE) 
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QUESTION  DATA  SHEET 


QuMtlon  Nuabsr  SCHCSA)  **  OOS 

OUESTIQN^  Thtt  davttlopnant  contractor  configuration  atatua  accounting 
procaduraa  can  ba  tranaitionad  to  or  ara  coapatibla  with  the 
.operation  aupport  activity  planned  configuration  atatua  accounting 
procaduraa . 

ACTIVITY  <;S) ;  Davalopaant  Contractor 

‘Tha  CRISP  and  tha  0/S  CMP  (or  tha  CRLCMP)  daacriba  the 
operation  aupport  planned  configuration  atatua  accounting  procaduraa, 
uaually  in  aceordanca  with  0oD-*STD-‘4S0A  and  HIL~STD**483A.  Tha 
operation  aupport  activity 'a  internal  configuration  atatua  accounting 
procaduraa  ahould  be  docuaentad  in  a  Software  Configuration 
Manageaant  Plan  or  an  aaaociatad  aet  of  procaduraa. 


SkSSSmi 


SaSEQliSS-ISSISUSlISfiSi 


RESPON?^  RATIONALE: 


7cOKPL3TiL7“5lSACRii”»  1,2.3.4.5.S  •  COKPLSTSLY  AGREE) 


QUESTION  DATA  SHEET 


QuMtion  Nu»b*r  SCH(SA>  -  009 

Th«  davslopmant  contractor  automated  support  tools  for 
configuration  status  accounting  of  basslinss  and  intsrnal  dsvslopasnt 
identifications  are  sdsquats. 

\  Dsvslopasnt  Contractor 

EXPLANATIONS :  The  automated  support  tools  might  include  a  control 
library »  automated  procedures  for  form  generation  end  retrievals 
automatic  traceability  for  version  control*  implemented  changes  and 
outstanding  problem  reports.  Traceability  of  requirements s  design, 
code,  end  teat  elements,  is  important  for  keeping  track  of  precise 
configuration  identification  of  baseline  data.  The  use  of  automated 
tools  is  essential  for  complex  software  systems. 


GLOSSASY : 


RESPONSE  INSTNUCTIONS; 


RESPONSE  SCORE:  _ 

“(COMPLETiLV'DISAGRsi  ■  1,2, 3, 4, 9, 6  ■  COMPLETELY  AGREE) 
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QUESTION  DATA  SHEET 


QuMtlon  Numbar  SCM(SA> 


01 


£USSXISHl  davalopaant  contractor  aoftwaro  configuration  atatua 
accounting  forma  ara  adaquata. 


4CTIX^TY<S> ;  Davalopnant  Contractor 

gjCg^^gATIQNg;  Tha  atatua  accounting  forma  auat  provide  adaquata 
information  to  track  internal  davalopmant  baaalinaa  aa  wall  aa  tha 
Functional,  Allocatad.  and  Product  baaalinaa.  HZL*ST0-4S2A  containa 
configuration  atatua  accounting  data  alamanta  and  ralatad  faaturaa. 
Typical  govarnmant  forma  include  ECP,  SCN,  configuration  control 
board  directive,  time  compliance  technical  order,  deficiency  report, 
and  changa/modif ication  report.  Internal  atatua  accounting  forma  muat 
bo  adaquata  to  track  nacaaaary  atatua  reporting  auch  aa  problam 
analyaia,  aolution,  change  implamantation,  and  cloaura.  Zn  addition, 
general  raporting  documanta  auch  aa  a  product  atatua  report,  open 
aoftwara  problama  report,  and  deliverable  document  atatua  report  muat 
be  maintained  in  order  that  contractually  raquired  atatua  information 
can  be  adequately  derived  and  juatified. 

SirSSSiSYi 


?SS?CNSS_^2IST3UCTI2N2i 


5f5SSl!2f-SAII2NAtfi 


RESPONSE _3C0Rg; 


QUESTION  DATA  SHEET 


QuMtlon  Number  SCH(SA>  -  Oil 

QUESTION ;  Subcontractor  configuration  Itaa  configuration  atatua 
accounting  procaduraa  ara  aonitorad  by  tha  davalopaant  contractor. 


ACTIVITY Davalopaant  Contractor 

B^PWftyATTQWS :  If  thara  ia  a  aubcontr actor*  it  will  ba  nacaaaary  that 
tha  davalopaant  contractor  raquira  configuration  atatua  accounting 
practicaa  aiailar  to  thoaa  raquirad  by  tha  procuraaant  activity.  If 
thia  la  not  dona*  than  tha  davalopaant  contractor  will  bo  raquirad  to 
ratrofit  tha  configuration  atatua  accounting  achaaa  of  tha 
aubcontractor .  Tha  configuration  atatua  accounting  idantif ication 
practicaa  of  tha  aubcontractor  nuat  ba  carafully  monitor ad  to  aaaura 
compatibility. 


GLOSSARY: 


A/6:  No  aubcontractora  ara  involvad  with  producing  aoftwara 
configuration  itama  for  tha  davalopaant  contractor. 


RSyPpNgE , RATION^^g : 


5r§£91J§5_§SSSIi _ 


A1-1S7 


QUESTION  DATA  SHEET 


QuMtlon  NuBlMr  SCMCSA)  -  012 

SUSSIIS^l  Status  of  aoftwaro  configuration  Itaaa  which  laplamant 
aafaty  provialona  ia  adaquataly  raportad. 

ACTIVITY <S) ;  Oavalopnant  Contractor 

KEfcASAIIQWS :  Configured  itaaa  which  iaplaaant  aafaty  provialona  ara 
fraquantly  controllad  by  aoftwara.  Status  of  this  aoftwara  auat  ba 
adaquataly  nonitorad  and  raportad  aa  aff acting  aafaty*  Safaty 
provisions  ara  cloaaly  ralatad  to  tha  raliability  of  aiaaion  critical 
coaponants*  aafaty  of  aiaaion  paraonnal*  nuclaar  affaeta*  and  so 
forth . 


GLOSSARY: 


Sa5£25I§i-I5iSISlISISSS21 

A/S:  Thara  ia  no  raquiramant  for  aafaty  provisions  to  ba 
iaplaaantad  or  controllad  aoftwara 


REggONSE„gATIONALE : 


RESPOMSE^SCOREi 

“<c5KPLiTELY’’DISAGRii'’“l,2,3,4,5,6  ■  COMPLETELY  AGREE) 


Ai’iea 


QUESTION  DATA  SHEET 


QuMtlon  Nuabar  SCn(SA>  -  013 

QUESTION;.  Status  of  aoftwara  eonfigurad  Itaaa  which  iaplaaant 
coaputar/coaaunleationa  aacurity  provlaiona  la  adaquataly  raportad. 


ACTIVITY <3)  :  Oavalopaant  Contractor 

Software  which  iaplaaanta  conputar/coaaunleation 
security  ia  particularly  important.  Any  auch  aoftwara  itaaa  nuat  be 
adaquataly  controlled  aa  part  of  the  trusted  coaputar  baaa.  If  the 
configured  aoftwara  ltaa<a>  are  thaaaalvaa  claaaifiad,  than 
appropriate  aacurity  labels  auat  be  attached  according  to  Air  Force 
labeling  raquiraaanta  and  accaaa  control  of  auch  itaaa  nuat  be 
enforced.  Statue  Inforaation  on  auch  software  auat  ba  reported. 


StSSSAHYi 

49ff*T,  *^^0  totality  of  threats,  vulnerabilities, 
and  protection  aechaniaaa  involved  with  dataraining  whether 
coaputar/eoaaunicationa  aaaeta  can  be  coaproaiaad  through  data, 
proceaa.  or  abuse  violations.  Security  provisions  exist  across  the 
adainistrative.  ayatea.  and  facility  categories. 


5r5222!Si-I25ISiiSIISN5i 

A/6:  There  ia  no  requirenant  for  security  provisions  to  be 
iaplaaented  in  software. 


glJPONSE^RATIONALfi 


RSSPOMSS  SCORER 

"<C0MPLETELY“DlSAGRii”»"l.2.3.4.S.6  ■  COMPLETELY  AGREE) 


i 


QUESTION  DATA  SHEET 


QuMtlon  NuMb«r  SCI1(SA>  >  014 

OySSTTON;  Th«  configuration  status  accounting  raqulraaanta  for  post 
doployaant  support  ara  adaquataly  addraaaad  In  tha  CRLCMP  (CRISP  and 
0/S  CMP}. 

6STiyiI]fiS2l  Op«iratlon  Support 

CRISP  and  0/S  CHP  (or  tha  JLC  Coaputar  Raaourcas 
Llfa  Cyela  Hanagaaant  Plan  -  CRLCMP)  ara  tha  kay  planning  docuaanta 
for  oparatlon  aupport  configuration  aanagaaant.  Tha  CRISP#  0/S  CHP* 
and  CRLCMP  ara  Intandad  to  ba  living  docuaanta*  avolvlng  to  provlda  a 
currant  vlaw  of  tha  configuration  aanagaaant  faaturaa  along  with  tha 
evolution  of  tha  ayataa.  Tha  CRISP  (firat  varalon)  la  raqulrad  aarly 
In  tha  llfa  eycla*  at  laaat  prior  to  full  acala  davalopaant.  Tha  0/S 
CHP  la  raqulrad  prior  to  tha  and  of  tha  full  acala  davalopaant.  Kay 
to  adaquacy  la  tha  coapatlblllty  of  tha  oparatlon  aupport 
configuration  status  accounting  and  tha  davalopaant  contractor 
configuration  status  accounting. 


RSSPONSI^INSTRUCTJONgi 


RESPONSE  RATIONALE; 


RESPONSE  SCORE: 


»  rCHPLETELV  AGREED 
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QUESTION  DATA.  SHEET 


QuMtion  Number  SCH(SA>  -  015 

QUESTIQN^.  The  operetlon  support  conf  Iguratlon  status  accounting 
procedures  ere  adequately  defined  to  handle  software  change  reporting 
requirements. 

Operation  Support 

CRISP  end  0/S  CMP  (or  the  JLC  Computer  Resources 
Life  Cycle  Management  Plan  *  CRLCMP)  are  the  key  planning  documents 
for  operation  support  configuration  aanegament.  The  CRISP.  0/S  CMP. 
end  CRLCMP  are  intended  to  be  living  docu.«enta.  evolving  to  provide  a 
current  view  of  the  configuration  management  features  along  with  the 
evolution  of  the  system.  The  configuration  status  accounting 
procedures  along  with  specific  responsibilities  should  be  defined  in 
the  CRISP  and  the  0/S  CMP^  It  is  not  enough  to  indicate  that  a  given 
directive,  regulation,  standard,  or  guideline  will  be  followed. 
Specific  detail  as  to  the  format  and  content  of  status  reports, 
organizational  structure.  Interface  responsibilities,  and  so  forth 
should  be  included  in  the  operation  support  Software  Configuration 
Management  Plan. 


RESPCNss  instructions: 


RESPONSE  .  rationa;.e  ! 


< COMPLETELY  DISAGREE  ■  1.2. 3. 4. 5. 6 


COMPLETELY  .AGREE: 
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QUESTION  DATA  SHEET 


QuMtion  KuBb*r  SCH(SA>  *  016 

autoMatad  aupport  toola  for  poat  daployaan^  aupport  of 
configuration  atatua  accounting  ara  adaquataly  addraaaad  in  tha 
CRLCHP  (CRISP  and  0/S  CMP). 

tSlDUmSli  Oparation  aupport 

Ay  AT  ?  ?  ■?  CRISP  and  0/S  CMP  (or  tha  JLC  Coaputar  Raaoureaa 

Lifa  Cycla  Hanagaaant  Plan  *  CRLCMP)  ara  tha  kay  planning  docunanta 
for  oparation  aupport  configuration  nanagaaant.  Th«  (HtlSP,  0/S  CMP» 
and  CRLCMP  ara  intandad  to  ba  living  docuaanta*  avolving  to  provida  a 
currant  viaw  of  tha  configuration  nanagaaant  faaturaa  along  with  tha 
avolution  of  tha  ayatan.  Tha  uaa  of  automated  aupport  toola  during 
davalopmant  and  tranaition  of  thoaa  taola  to  uaa  during  poat 
daploynant  aupport  ia  an  important  canaidaration  for  tha  overall 
anhancamant  of  aoftwara  aupportability *  Tha  lack  of  auch  toola  to 
manage  tha  configuration  atatua  aeaaunting  of  tha  varioua  baaalinaa 
ahould  ba  conaidarad  a  daficiancy* 

GLOSSARY : 


e 


RSSPCNSS .INSTRUCTIONg; 


RESPONSE  RATIONALE: 


RESPONSE  SCORE: 


* 

«»  «  W  •  ^  #-w 


•«*^VOT  V 


4 


.a  » 


SOFTWARE  CONFIGURATION  MANAGEMENT  AUDIT/REVIEU 


Th«  quMtiona  SCM<AR}-001  through  SCM(AR)>013  «ddr««a  lasuas  of 
•oftwara  configuration  Audit/Raviaw  for  tha  proeuranant*  davalopnant 
contractor »  and  oparation  aupport  activitiaa.  Softwara  Configuration 
Audit/Raviaw  ia  conductad  to  varify  that  a  coaplatad  aoftwara  product 
aatiafiaa  raquiraaanta.  Procuraaant  conducta  official  contractual 
configuration-oriantad  audita  and  raviawa.  Portiona  of  tha 
davalopaant  raviawa  <PDR»  COR#  Taat  Raadinaaa  Raviaw(TRR)}  ara 
davotad  to  conf iguration-oriantad  raviaw  of  production  producta  aa 
identified  in  davalopaantal  baaalinaa.  Tha  aa^or  configuration  audita 
for  procuraaant  ara: 

Cl>  Functional  Configuration  Audit  (FCA> 

(2)  Phyaical  Configuration  Audit  (PCA> 

A  Formal  Qualification  Raviaw  (FQR>  for  each  CSCI  conatitutaa  a 
configuration  audit  if  it  ia  included  aa  part  of  tha  CSCI 
Configuration  Index.  In  thia  caaa  procuraaant  and  parhapa  oparation 
aupport  rapraaantativaa  raviaw  tha  product  apacif icationa, 
Praliainary  Qualification  Taat  (PQT>  data#  and  Formal  Qualification 
Taat  (FQT)  data  to  certify  that  tha  CSCI  ia  qualified  for  ita 
intended  application.  T>ta  FQR  followa  tha  FCA  and  PCA. 

The  davalopaant  contractor  activity  partieipataa  in  tha 
preparation  for  and  conduct  of  foraal  configuration  audita  and 
raviawa.  In  addition#  inttrnal  audita  and  raviawa  are  typically  part 
of  a  broad  quality  aaauranca  function. 

Tha  oparation  support  activity  haa  a  reaponaibility  to  monitor 
procuramant  and  davalopmant  contractor  audita  and  raviawa.  This 
monitor  information  ia  integrated  into  tha  oparation  support  plana. 
Tha  oparation  aupport  configuration  audit/reviaw  ia  similar  to  tha 
procuramant  for  tha  foraal  baaalinaa  and  like  tha  davalopmant 
contractor  for  tha  internal  audita  and  raviawa. 


Al-193 


QUESTION  DATA  SHEET 

QuMtion  Number  SCK(AR>  -  001 


£USSXI2^1  procurement  policy#  mtenderde#  end  conventions  epplied 
to  the  eudit  end  review  of  eoftwere  conf iguretion  items  ere  edequete. 


ACTIVITY <S) t  Procurement 

Audit/r  eview  of  computer  eoftwere  configured  items  ie 
described  in  directives#  regulstions#  end  guidelines:  APR  65*3#  APR 
S7-4#  APSC?  600-3#  OoOD  5000.29#  APR  800-14#  DoOD  5010.21#  APSCP 
600-7.  It  is  the  responsibility  of  the  procurement  ectivity  to  essure 
that  proper  policy#  etenderds#  end  conventions  ere  required  for  the 
eudit/review  of  configured  items.  Contreetor  complienca  documents 
which  could  be  requized  include  HXL-STD-480A#  HIL-8T0-482# 
KIL-STD-463A,  N1L-ST0-490A#  N1L-STD-1521B#  end  DoD-STD-2167 . 


OLOSSARYt 

SiiisfSEl— SfiCiiSlilSilfiS— JiSMs  Softwere  elements  which  era 
designated  for  Configuration  Audit/Review  by  the  contractual 
requirements. 

The  process  of  informal  end  formal  verification 
that  ^e  particular  product  has  satisfied  e  specified  set  of 
requirements.  ^ 


SSSSQSSS^ItiSISUSIISSSl 


REgPQN??  RAT|9NALg : 


RESPONSE^SCORS^ 

“7c0?i?LETiLY”5lSAGRii’’~l,2,3,4,5,6  ■  COMPLETELY  AGREE) 
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QUESTION  DATA  SHEET 


QuMtion  Numbar  SCK(AR>  -  002 

fiUSSXIQlil  proeuraaittnt  activity  haa  laplaaantad  adaquata  aoftwara 
configuration  audit/raviaw  baaad  upon  ragulationa  to  control  tha 
functional  and  phyaical  character latica  of  all  CSCZa. 

ASIiyilXiSli  Procuraaant 

Tha  procuraaant  program  managar  ia  raaponaibla  for 
Implananting  a  configtiration  managamant  program  baaad  on  AFR  65-3 
that  will  identify#  document#  and  control  tha  functional  and  phyaical 
character lat lea  of  all  CSCIa  under  davalopaant.  Primary  planning 
document  ia  tha  Program  Kanagamant  Plan  (PHP>.  Othar  activitiaa 
include:  coordinating  raqulramanta  with  uaing  and  aupporting 
aganciaa;  reviewing  contractor  plana;  auditing  contractor 
implamantation  of  plana;  anauring  configuration  idantif Icationa  for 
all  CSCZa  are  proparly  documented;  controlling  anginaaring  changaa  to 
baaalinaa;  providing  intarfaca  control  for  diatribution  of  changaa; 
and  preparing  tha  PHRT  package  for  tranafar  to  tha  operation  aupport 
activity. 

^SSAPYl 


RESPONSE  INSTRUCTIONS; 


RESPONSE  RATZONAI.E; 


’’7c5j!?iIiTiLY'5:sAGRii"« 


1.2.3.4.S.S 


COMPLETELY  AGR'SE) 
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ia»am»aiiirMHaniinaBMaftanMiinQfloanftBftfyvuuuwvwuinaniinmvvifuvcrvwT.» 


QUESTION  DATA  SHEET 


QuMtion  Ntijib«r  SCM(AR)  -  003 

^SSHSm  procur«m«nt  conf iouratlon  Managcaant  planning  docunanta 

contain  auf aidant  guidance  for  configuration  audlt/ravlaw. 

T  Procuraaant 

siSEi^misssj,  Tha  aa^or  planning  docuaanta  for  procuraaant  ara  tha 
Program  Kanagaaant  Plan  (PHP>»  tha  Raquaa't  for 

Propoaal (RFP) /Statanant  of  WorkCSOU)^  tha  Contract  Data  Raqulraaanta 
LlatCCDRD*  and  tha  Coaputar  Raaourcaa  Intagratad  Support  Plan 
(CRISP) .  Tha  Joint  Loglatlca  Coaaandara  aoftwara  atandardlzatlon 

program  haa  a  raplacamant  for  tha  CRISP  called  tha  Computer 

Raaourcaa  Life  Cycle  Hanaganant  Plan  (CRLCHP).  AFR  600-14  calla  for 
tha  Inclualon  of  configuration  managamant  concapta  In  tha  PKP 
Including  apaclf Icatlon  and  Intarfacaa) .  Tha  RFP/SOW  daf Inaa  tha 
exact  acopa  of  tha  davalopmant  contractor'a  configuration 

audlt/ravlaw  raaponalbllltlaa.  Tha  CDRL  Idantlflaa  all  dallvarabla 
data  Itama  including  CSCIa  whldh  tha  davalopmant  contractor  muat 
deliver  and  control.  Tha  CRISP  ia  to  Include  aaaignmant  of 

configuration  audlt/ravlaw  raaponalbllltlaa  during  poet  deployment 
with  detailed  procaduraa  defined  In  tha  Oparatlonal/Support 
Configuration  Hanagamant  Procaduraa  <0/S  CMP)  • 

swassi-sii 


RESPONSE  INSTRUCTIONS: 


RESPONSE  .RATIONAIfS: 


RESPONSE  SCORE; _ : _ 

”7c5KPLETELY”DiSAGRii”«”l,2,3,4,S,6  ■  COMPLETELY  AGREE) 
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QUESTION  DATA  SHEET 


QuMtlon  Nuabar  SCH<AR>  -  004 

QUESTION :  Th«  conduct  of  foraal  ravlawa  and  audita  follows  a  forsat 
baaad  on  tha  chackllats  froa  HIL*STD-1521B»  sppropriatsly  tallorad 
for  tha  apacific  aoftwara  audlt/ravlaw. 

Procuraaant 

fygWAWATIOWS;  Tha  MIL-STD-1S21B  la  tha  eoapllanea  docuaant  for 
davalopaant  contractor  audita  and  ravlawa.  It  la  procuraaant '  a 
raaponalblllty  to  provlda  tha  guidanea  for  what  atandarda. 
ragulatlona*  and  tailoring  guidance  la  raqulrad.  It  la  tha 
davalopaant  contractor's  raaponalblllty  to  follow  tha  raqulraaanta. 
Thara  ara  ralatad  configuration  audita  and  evaluation  cheek  lists 
(a.g.,  FCA  and  PGA  preparation  eheekllata  In  HZLoST0-1521B»  ECP 
praparatlon  chackllats  In  DoD-STD’‘480A  and  aodlflad  by  HIL*STD>483A» 
Coaputar  Raaourca  Hanagar'a  Checklist  baaed  on  APR  800-14.  and 
attaehaanta  3.4.S  of  AFSCP  AOO-7  on  RFPa  and  eontraeta) . 

glossary; 


RESPONSE  INSTRUCTIONS: 


RESPONSE  RATIONALE; 


QUESTION  DATA  SHEET 


QuMtion  NuBb«r  SCM(AR)  -  005 

ftSEsnsNi  Th«  Boftwar*  product  accoptaneo  raqulramanta  ara  adaquata. 


tSUimUSll  Procuraaant 

Tha  aoftwara  product  aceaptanea  erltarla  ahould  ba 
elaarly  doe^aatad.  Tha  aceaptanea  taata»  daaonatratlona*  0T6E,  OT&E, 
qualification  taata*  audita#  and  raviawa  all  fora  a  part  of  thaaa 
aceaptanea  raquiraaanta .  Tha  procuraaant  activity  haa  tha 
raaponaibility  to  aaka  aura  aueh  aceaptanea  raquiraaanta  ara  eoat 
a£faetiva#  fxinetionally  adaquata#  and  apaeifiad  fron  tha  tina  of  tha 
RFP/SOW/CDRL.  Fraquant  aodifieation  to  tha  original  raquiraaanta 
indieataa  a  lack  of  undaratanding  eonearning  tha  original  ayatan 
apaeifieationa.  Thia  ia  likaly  to  raault  in  a  laaa  aatura  ayatan  to 
ba  aupportad. 

GLOSSARY; 


/itv 


Rg2£2MSg^IN5IRUCTi0M2i 


RESPONSE  RATIONAL^; 


'*c5KPLiTiLY“0ISAGRii”«”l.2.3.4.5.6  -  COMPLETELY  AGREE) 
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QUESTION  DATA  SHEET 


QuMtion  Nuabsr  SCMCAR)  **  006 

OyE^TIONl  Th«  davalopaiMt  contractor  Intarnal  configuration 
audit/raviaw  procaaa  facilitataa  tha  davalopaant  of  high  quality 
production,  aoftwara. 

AyriVITycg) ;  Oavalopaant  Contractor 

Tha  intarnal  davalopaant  contractor  procaduraa  for 
audit  and  raviaw  can  ba  an  important  part  of  tha  procaaa  to  build-in 
aoftwara  aupportability  charactariatica  in  tha  aoftwara  producta. 
Thia  ahowa  up  in  both  tha  tranaition  of  lifa  cycla  procaaaaa  and  in 
tha  tranaition  of  tha  aoftwara  product  baaalina.  Tha  intarnal 
audit/raviaw  procaaa  alao  tanda  to  raflact  how  auccaaaful  tha  formal 
contractual  audit/raviawa  will  ba. 


St2§34S2i 


2ES29i!Ss.II!SI3!!SII3liSl 


3ESP0MSE_SATigNALg^ 


( CZJi?\Z“‘zLy  ZZSAGtiEE  «  1 


^  ^  d  ^  ^ 


CCy.PLSTzLy  AGPSE) 
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QUESTION  DATA  SHEET 


QuMtlon  Nuab«r  SCH(AR}  ~  007 

9UEST^^QN;  Conf Iguratlon  audlt/rttvi^w  IntarfacM  Mong  procur«iii«nt , 
d«v«lopK«nt  contractor#  and  oparation  aupport  actlvltlaa  ara 
adaquata. 

ACTIVITY *  Davalopaant  Contractor 

aetlvitiaa  raquira  .ln:forBatien  £roB  all  lavala  of 
tha  audlt/raviaw  proeaaa  in  ordar  to  proparly  plan  for  apaclfic 
activity  raaoureaa#  funding  lavala*  raaolution  of  problaaa*  and  ao 
forth.  An  Intarfaea  control  working  group  ia  an  approprlata  aadiun 
for  coordinating  achadula*  raaponalbllitiaa*  contractual  aspacta*  and 
raaulta  of  tha  audit/raviawa . 


GLOSSARY; 


RESPONSE  instructions: 


RESPONSE  RATIONALE; 


RESPONSE  score;  _ 

CCiipiIiTiL7”5l5AGRES  » 


4.3.3  »  rCMPLETsL?  .AGREE; 


QUESTION  DATA  SHEET 


QuM'tlon  Nuabttr  SCH(AR)  ~  008 

QUESTION ;  Th«  d«v«lopn«nt  eontraetor  configuration  aanagaaant  tool 
aupport'~faeilitataa  tha  audit/raviaw  of  tha  procaaa  by  which  changaa 
ara  incorporatad  into  configuration  idantificationa. 

j^<fT^VITy<S) :  Davalopaant  Contractor 

It  ia  raquirad  to  audit/raviaw  all  changaa  that  hava 
baan  incorporatad  into  a  configuration  identification.  It  graatly 
facilitataa  tha  audit/raviaw  procaaa  if  tha  change  procaaa  ia 
autoaatad  and  tool  aupport  ia  available  to  indicate  tha  configuration 
identification  with  and  without  tha  incorporatad  changaa. 
Configuration  identification  coaparator  toola  can  indicate  which 
alaaanta  of  tha  configuration  idantif Ication  hava  bean  changed  aa  a 
confirmation  of  tha  incorporatad  changaa.  Tha  availability  of  auch 
automated  tool  aupport  graatly  facilitataa  tha  efficiency  and 
accuracy  of  tha  audit  and  review  activity. 

GLOSSARY; 


REyowgg_iNgTgucTigNS; 


55SEQ5iSE_RATiOHAti: 


SCCSS ! 

<CCKru£T£LY  OXoAGctEc  «  la2aOa4aSa6  «  COnPLETZLY  AGZEE) 


QUESTION  DATA  SHEET 


QuMtion  Nu»b«r  SCH(AR)  -  00 


fi&SSXlQSil  Subcontractor  conSigurablon  Itan  audit/ravlaw  practicas  ara 
aonitorad  by  tha  davalopaant  contractor  • 


^JlXVJXXiSii  Davalopaant  Contractor 


thara  ia  a  aubcontractor ,  it  will  ba  nacaaaary  that 
tha  davalopaant  contractor  raqulra  configuration  audit/raviaw 
practicaa  aiailar  to  thoaa  raquirad  by  tha  proeuraaant  activity.  Z£ 
thia  ia  not  dona*  than  tha  davalopaant  contractor  will  ba  raquirad  to 
ratrofit  tha  audit/raviaw  achaaa  of  tha  aubcontr  actor.  Tha 
audit/raviaw  practicaa  of  tha  aubcontractor  auat  ba  carafully 
aonitorad  to  aaaura  coapatibility. 


GLOSSARY : 


A/6:  No  aubcontractora  ara  involvad  with  producing  softwara 
configuration  itama  for  tha  davalopaant  contractora. 


RESPONSE  rat;onale  : 


RESPgNSS_SCgREi 

”” *  l.Z.2.  7.2.3  *  2CHPLETEL?  AG3EI 
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QUESTION  DATA  SHEET 


QuMtion  Nuxbttr  SCH<AR}  -  010 

QUESTION;  Conflgursd  itM«  which  laplM«nt  provisions  srs 

sdsqus-bsly  suditsd  snd  rsviswsd. 

ACTIVIXYi^lL  Osvslopasnt  Contractor 

S?^PLAyATIQWS;  Configured  itsss  which  iaplsssnt  safety  provisions  sra 
frequently  controlled  by  eoftwere*  This  software  suet  be  sduqustely 
identifiud  ss  effecting  eefety.  Safety  provisione  ere  cloeely  related 
to  the  reliability  of  eieeion  critical  coaponente*  safety  of  aission 
peraonnel*  nuclear  effects*  snd  eo  forth* 


RESPONSE  INSTRUCTIgNS ; 

A/$;  Thera  Is  no  requirenent  for  safety  provisions  to  be 
iaplaaantad  or  controlled  by  software 


RSSPpN^g  RAT|0NAVE; 


RESPONSS^SCORE; _ 

"(COMPLSTELy’dISAGREE  •  1*2*3*4*S*6  «  COMPLETELY  AGREE) 


A1 -203 


QUESTION  DATA  SHEET 


Question  Nusbsr  SCHCAR)  -  01 


fiU&SXIQlil  Software  configured  Iteee  which  implement 
computer /communications  security  proviaione  ere  adequately  audited 
end  reviewed. 

/19TyVITY<^) :  Developaent  Contractor 

gXPtrAKATXOWS  *  Software  which  ‘  impleaente  eoaputar/communlcation 
security  is  particularly  isportent.  Any  much  eoft%#ere  items  must  be 
edequetely  audited/reviewed  ee  pert  of  the  trusted  computer  baas.  If 
the  configured  software  item<e>  are  themeelvee  cleeeified*  then 
approprieta  security  labels  must  be  attached  according  to  Air  Force 
labeling  requiramente .  Adequacy  of  such  labeling  procedures  should  be 
audited/ reviewed . 

G^gSSARYj 

j  totality  of  threats,  vulnerabilities, 
end  protection  mechsnisme  involved  with  determining  whether 
computer /communications  assets  can  be  compromised  through  data, 
process,  or  abusevioletions.  Security  provisions  exist  across  the 
administrative,  system,  and  facility  categories. 


A/6:  Thera  is  no  requirement  for  security  provisions  to  be 
implemented 

in  software. 


RESPONSE  RATZOMALE: 


RE5P0NSg_SCgREi _ 

(c5nPLETiLY~0ZSAGRii'«  1,2,3,4,S,6  >  COMPLETELY  AGREE) 
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QUESTION  DATA  SHEET 


QuMtlon  Nu»b«r  SCH(AR>  -  012 

fiUiSIISlil  Th«  software  configuration  audit/raviaw  raquiraaanta  for 
poat  dapToyaant  aupport  ara  adaquataly  addraaaad  in  tha  CRLCHP  (CRISP 
and  0/S  CMP> . 

*  Operation  aupport 

KgWAIfATIQWS;  Tha  CRISP  and  0/S  CMP  (or  tha  JLC  Coaputar  Raaourcaa 
Lifa  Cycle  Hanagaaant  Plan  -  CRLCMP)  ara  tha  kay  planning  docuaanta 
for  operation  aupport  configuration  aanagaaant.  Tha  CRISP,  0/S  CUP, 
and  CRLCMP  ara  intandad  to  be  living  docuaanta,  avolving  to  provide  a 
currant  view  of  tha  configuration  aanagaaant  faaturaa  along  with  tha 
evolution  of  tha  ayat'aa.  Tha  CRISP  (firat  varaion)  ia  required  early 
in  tha  lifa  cycle,  at  laaat  prior  to  full  acala  davalopaant.  Tha  0/S 
CMP  ia  required  prior  to  tha  and  of  tha  full  acala  davalopaant « 

glossary; 


RSS&QNSS.INSI5SSIISliSl 


R^ifpqyss  rationale; 


RESPONSE  SCOREJ. _ 

'TcShPLEtIly'dISAGREE  ■  1,2, 3, 4, 5, 6  >  COMPLETELY  AGREE) 


OUSSTXON  DATA  SHEET 


QuMtion  Nuab«r  SCH(AR}  -  013 

QUISTIOW!  Th«  automatad  aupport  toola  for  poat  daployaant  aupport  of 
confisuration  audlt/ravlaw  ara  adaquataly  addraaaad  in  tha  CRLCHP 
(CRZSP  and  0/S  CMP). 


(^yy|VITY<S) ;  Oparation  Support 

^XPLAMATIOMS;  Tha  CRISP  and  0/8  CMP  (or  tha  JLC  Coaputar  Raaourcaa 
Lifa  Cyela  Kanaqaaant  Plan  -  CRLCMP)  ara  tha  kay  planning  docuaanta 
for  oparation  aupport  configuration  aanagaaant.  Tha  CRZSP «  0/S  CMP, 
and  CRLCMP  ara  intandad  to  ba  living  docuaanta,  avolving  to  provida  a 
currant  viaw  of  tha  configuration  aanagaaant  faaturaa  along  with  tha 
avolution  of  tha  ayataa.  Tha  uaa  of  autoaatad  aupport  toola  during 
davalopaant  and  tranaition  of  thoaa  toola  for  uaa  during  poat 
daployaant  aupport  ia  an  iaportant  conaidaration  for  tha  ovarall 
anhancaaant  of  aoftwara  aupportability.  Tha  lack  of  auch  toola  to 
aaaiat  in  tha  audit  and  raviaw  of  tha  varioua  baaalinaa  ahould  ba 
conaidarad  a  aarioua  daficiancy. 


2SSSSIiSS^SSI22SIIQSli 


RSgPQNgS^gCORg;  _ 

"(c5KPLSTELY“DISAGRii"»'’l,2,3,4,3,6  ■  COMPLETELY  AGREE) 
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MnwMnflPflhflnannt  lainnaruinflf 


THE  BOM  CORPORATION 


oun/«o  1  ^ 


ATTACHMENT  A2 


SUMMARY  LIST  OF  QUESTIONS 


This  ittichunt  contiins  a  susaiary  list  of  ill  Softwara  Lift 
Cyclf  Proctss  qufstlons.  Tht  questions  art  listed  In  the  order  In 
which  they  apptir  In  Attachment  Al. 


A2-1 


SOFTWARE  PROJECT  MANAGEMENT  PLANNING 


SPM(Plf^-<294L,9!iSSTIQMt  Planning  for  eonputor  rosourcos  has  boon 
odoquoto  with  roopoct  to  ocquioition*  dovolopnont^  loGlotlco*  ond 
troining. 

,.9yf?TT9N:  Procurooont  plonning  for  cooputor  rooourcos 
hoo  boon  conolotont  with  tho  oyotoo  dovolopnont  ond  ocquloitlon  plan. 

j^PMCPL)  -OOg^  Plonning  for  cooputor  roooureoo  hoo  boon  boood 
upon  on  ocquloitlon  ochodulo  with  odoquotoly  opoclflod  allootonoo. 

3y^<.gL) Cooputor  rooourcos  hovo  boon  odoquotoly 
oddrooood  oo  oojor  conoldoratlono  ot  procurooont  roviowo#  oudlto.  ond 
oonogooont  ovoluotlono. 

S£2!i£^2r9SSi.9y£2J12jfi  Plonnod  cooputor  rooourcos  hovo  boon  onolyzod 
odoquotoly  by  onouro  eonforoonco  with  ototod  oporotlonol  ond  support 
roqulrooonto. 

?PBi?HrS9§,*-9yS?XIQy.ji  Procurooont  plonning  for  ooftworo  quality 
ottrlbutoo  hoo  boon  odoquotoly  oophoslzod  throughout  tho  ooftworo 
llfo  cyelo  ocquloitlon. 

*QO7j^qyE?TI0N:  Morglno  for  rosorvo  cooputor  rooourco  copoclt#Sit 
to  provide  for  lotor  product  loprovoaonto  oro  odoquoto. 

SSS  L?\t  ?.  I ,  ,9.y  Accoptoblo  tochnlquoo  hovo  boon  uood  to 
ootlaoto  ond  monitor  ooftworo  costs  throughout  tho  systom  llfo  cyclo. 

5£2iE^*2.;222l-Sy55IIfi!£i  Tho  CRLCMP  <CRISP,  0/5  C»P)  contains  odoquoto 
spoclf  leotlons  of  tho  acquisition  roqulrononts  for  computor 
rooourcos. 

5£I!i£t2j2i2i— SyfSIISEi  Tho  CRLCMP  (CRISP,  0/5  CMP)  odoquotoly 
oddrossoo  tho  rosponolbllltios  ond  prododuroo  to  onouro  propor 
ooftworo  configuration  oonogooont  throughout  tho  oyotoo  llfo  cyclo. 

r911  j_9V6STI!?l?i  Tho  project  oonogooont  rosponslbillty  for 
Intogroting  cooputor  rooourcos  Into  o  systoo  has  rooslnod  contra llzod 
throughout  tho  llfo  of  tho  oyotoo. 

?£ff^EW2lS12i--9V5?TI9y-T  Tho  CRWG  organ  Izot  ion  hoo  boon  odoquoto 
throughout  tho  oyotoo  llfo  cyclo. 

5£SIiEi»l-21.2l— .fiUSSIIQlii  Tho  CRWG  has  had  cloorly  opoclflod 
rooponslbllltlos  ond  opproprloto  authority  to  loplooont  thooo 
rooponoibilitios  throughout  tho  systom  llfo  cyclo. 

^gM<g^) -93.5:  Tho  CRWG  has  properly  osourod  that  cooput«» 
rooourcos  comply  with  ostabllshod  policy,  procoduros,  plans, 


3PW(PL)-01g:  OUggTIQN^  Software  quality  aaaaaamant  procaduraa  hava 
baan  adaquataly  defined  to  aeet  aanageaent  pollrlea  and  appropriate 
ragulationa*  conform  to  atandarda,  and  meet  performance  and  quality 
raquiramanta  throughout  tha  ayatem  life  cycle. 

SPW<PL)»016;  QUESTION;  Planning  for  DT6E  of  computar  raaourcea  haa 
baan  adequate  throughout  the  ayatem  life  cycle. 

Plennlng  for  OTfcE  of  computer  reaourcea  haa 
been  adequate  throughout  the  ayatem  life  cycle. 

SPH<PL)-01S;  OUE^T^OW;  Software  atandarda  have  been  adequately 
apecified  throughout  tha  aoftware  life  cycle. 

Th*  plenning  for  organic  and/or  contractor 
support  during  poet  deploymant  aoftware  support  haa  bean  adequata. 

S^iEW2l922i _ MfiSlIQlil  Contractual  documents  hava  axpllcltly 

eatabllahed  Govarnmant  rights  to  all  computer  reaourcea  required  to 
develop#  operate#  aimulata#  teat#  and  support  tha  software. 

i EW2  r921 i - 9Vf STIOtfi  Plenning  for  risk  analyaia  to  identify  areas 
of  computar  raaource  risk  haa  been  adequate. 

2£Si.EW2l222l.9UlSXIfi!il  ^  miaalon/function  matrix  (or  equivalent) 
clearly  identifies  primary  functional  capabilitiea  to  ba  implemented 
by  the  aoftware. 

*— SUI5IISl!i  Plenning  for  interoperability  with  other 
ayatama  haa*’baan  adaquataly  addreaaad. 

SPt^epy.) -02^1  QUESTION;  Prior  to  each  ayatam  milastone*  intaraarvicing 
potential  and  life  cycle  coat  implicaticna  of  software  support 
options,  hava  been  appropriately  addressed. 

SBSi£^2r223i.9y£3Je91!2  ^^e  procurement  and  operation  support  planning 
documents  hava  been  adequately  updated  as  living  documents  throughout 
tha  system  life  cycle. 

‘^^e  principles  and  methodologies  provided  in 
the  regulations  hava  been  appropriately  incorporated  into  the 
software  test  and  evaluation  plane. 

Planning  for  systematic#-  quantitative#  and 
objectively  reportable  software  tests  has  been  adequata. 

Spy  (PL).  7028;  quSgT|0N^  Planning  for  sharing  of  software  test  results 
across  lifecycle  phases  and  among  test  organizations  has  been 
adequate . 

SPytPLl •029j_0yfSTigN_;  Tracking  of  computar  resource  utilization  has 
been  adaquataly  planned. 
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v.rl.n<^ 


gPS ( PL ) -030  z ,oyg?I^0N  ^  Th«  project  softwAra  budg«t/eoAt 
(budgatAd  •  ActuAl)  AppAArA  to  b#  roAAonAblA. 

SS2JiP^2-S31i-9y£SJI2!!!i  prOJACt  AOftWArA  AChAdulA/COAt  .  VAriAZlCA 

CeonAUAAd  *■  AchAdulAd)  AppAArA  to  bA  rAAonoblA. 

?yW<PL) ThA  coAt  And  AchAdulA  contrActUAl  rAportlng 
rAqulrAAAntA  AppAAr  to  bA  AdAquAtA. 


SOFTWARE  PROJECT  MANAGEMENT  ORGANIZATION  STRUCTURE 


?PH<;QS)  AoftwArA  rAqulrAAAntA  Nava  bAAn  AdAquAtAly 

AllocAtAd  to  AlAAAntA  of  A  Work  BrAAkdown  StructurA  (WBS> . 

SPM^,021-g9^1..gUg^XI<3N^  ThA  AoftwArA  rAlAtAd  tAAkA  or  a  elAArly 

IdAntifiAd  in  thA  WBS. 


•  ThA  kAy  projAct  pArAonnAl  And  thAlr 
AAAlgnAAntA  In  rnlAtlon  to  thA  WBS  AoftuArA  rAlAtAd  tAAkA  ArA  elAArly 
IdAntifiAd. 

SP8<QS) *00^ QUESTION*  ThA  coordlnotion  of  modlflCAtlonA  to  thA  UBS 
AAOng  All  ACtlVltlAA  hAA  bAAn  AdAqUAtA.  ^ 

5£3iS21l2S5i-9yi5IIQ5I  ThA  procurAWAnt  pArAonnAl  AtAfflng  hAA  hod 
continuity  throughout  thA  AoftwArA  Ilf a  eyelA  phAAAA. 

SPM (OS ).«006^,QUg§7tQN:  ThA  rAtlo  of  AxporlAncAd  procuromAnt  projAcc 
pACAonnAl  to  thA  total  numbar  of  pro^Act  parAonnAl  hao  baan  adaquata 
throughout  tha  acftwara  Ufa  eyela  phaaaa. 

5221251  Ji22Zi-3y55Z12iJ2  Tha  nuabar  of  procuraaant  paraonnal  haa  baan 
adaquata  throughout  tha  aoftwara  Ufa  eyela  phaaaa. 


SPM<OS)-008;  QUESTION:  Tha  davAlopaant  eontraetor  paraonnal  ataffing 
haa  had  eontlnulty  throughout  tCa  Aoftwara  Ufa  eyela  phaaea. 

SPM (OS) -QOS  t  Q^^^TIOW  T  Tha  ratio  of  AxparlaneAd  dAVAlopaant 
eontraetor  pro^aet  paraonnal  to  tha  total  nuabar  of  pro^aet  paraonnal 
haa  baan  adaquata  throughout  tha  ooftwara  Ufa  eyela  phaaaa. 

5S2i252r212i.2y55J121f2  Tha  nuabar  of  davalopaant  eontraetor  paraonnal 
haa  baan  adaquata  throughout  tha  aoftwara  Ufa  eyela  phaaaa. 

^22111 -925511231  Tha  oparatlon  aupport  paraonnal  ataffing  haa 
had  continuity  throughout  tha  ooftwara  Ufa  eyela  phaaaa. 

5ESi221l212l.92SSXI23l  Tha  ratio  of  axparlancad  oparatlon  aupport 
pro^act  paraonnal  to  tha  total  nuabar  of  pro^act  paraonnal  haa 
adacuaia  thrcuchcut  tha  aoftwara  Ufa  r/cia  phaaaa.  ’vlcv 
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SESiSSllSlSi-fiUSSIIQlii  Th«  number  o£  operation  eupport  personnel  hss 
been  adequate  throughout  the  software  life  cycle  phases. 

SPW<0S>-014;  OUESTIQW;.  The  Internal  interfaces  among  procurement 

organization  elements  have  been  adequate. 

9ygSTI0N;  The  internal  interfaces  among  development 

contractor  organization  elements  have  been  adequate. 

3PW<03)-016; _ The  internal  interfaces  among  operation 

support  organization  elements  have  been  adequate. 

OyiyriQ^^  The  procurement  phyaicai  organization 
structure  has  been  adequate. 

S£0ifiS2zSldi _ ■9VE?T?0^«  The  development  contractor  phyaicai 

organization  structure  has  been  adequate. 

3PI^(03> -01?:  0yS3TIQWL  The  operation  support  phyaicai  organization 
structure  has  been  adequate. 


30FTWAPE  PROJECT  MANAGEMENT  DE3IGN  METHODS 


The  procurement  design  analysis  studies  have 
provided  adequate  design  guidelines  for  the  development  contractor. 

$Eui.S.ull292l.SUSSXISil£l  The  standards  for  software  design  required  by 
the  procurement  activity  are  adequate. 


gPW<DM2;0$?^_QyggTjgy  l  The  software  design  methodology  used  by  the 
development  contractor  is  adequate. 

The  design  standards  and  methods  adopted  for 
use  by  the  operation  support  activity  during  post  deployment  software 
support  are  adequate. 

• 

The  System  Design  Review  process  has  been 

adequate . 


3??I(pM2200§i_..3Vg?JI9W ;  The  software  requirements  appear  to  be 
reasonable . 


§Ei;i2i;2jS2Zi_9i'S5ZI3tlJ,  "he  number  of  software  raquiramenta  which 

cannot  be  traced  to  an  end  item  product  is  minimal. 

aEXlsullSiSi-aiiiSXXSiIi  The  number  of  software  raquiramenta  which 

cannot  be  tested  are  minimal. 


_ OUfSIigNi  The  profile  of 

••  -5  r -iaacnabla . 


changes 


to 


software 
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5PXOM>  *010 i  .O^^STION;  Th«  profile  of  unroaolvod  aoftworo  rovloj 
•sctlon  Itoms  Is  rsssonsbls. 


SPXfpX) *011 i _ GUEST^QN;  Ths  dovslopsonb  contrsctor  rsquirsssnts 

snslysis  procsss  has  boon  sdsqusts. 

dsvslopssnt  contrsctor  top  Isvsl  dsslgn 

procsss  has  boon  adsquats. 

SSSSSSlzQiSl^BaSSIlSEi  dsvslopssnt  contrsctor  dstsllsd  dsslgn 

procssa  has  bssn  sdsqusts. 

Ths  dsslgn  cosplstlon  of  CSCZs  rslstlvs  to  ths 
softwsrs  Ilfs  cycls  dsvslopssnt  schsduls  has  bssn  rsssonsbls. 

OUgSTIOW;  Ths  dsvslopssnt  contrsctor  sonltor  of  ths 
subcontractor  software  dsslgn  procsss  hss  bssn  sdsqusts. 

S£!11.2!lll2X§I-9U£SXI&!il  design  specifications  for  ths  software 
products  contain  adsquats  Information  to  Isplsssnt  ths  software  with 
ths  required  functionality  and  within  ths  schsduls  and  budget 
rsqulrsssnts . 

operation  support  concept  for  dsslgn  of 
software  revisions  during  post  deployment  software  support  is 
adsquats. 

S£2Ji222l215i-Syi5XIS2!i  operation  support  concept  for  dsslgn 

review  during  post  deployment  software  support  Is  adsquats. 


SOFTWARE  PROJECT  HANAGEHENT  IMPLEMENTATION  METHODS 


5£i!iaS2j221i— SyS5XjCS2fi  procurement  activity  has  adequately 
monitored  ths  Isplsssntatlon  of  ths  softwsrs  dsslgn  specifications. 

SPiy  <  IM)  ~09^t  _.SK2XmQSL  procurement  test  orgsnlzatlon  Interface 
with  ths  dsvslopssnt  contractor  Is  sdsqusts  enough  to  assure  success 
of  ths  system  Integration  tests. 

ggW<{M)*99?T  operation  support  activity  has  bssn 
actively  Involved  with  ths  dsvslopssnt  contractor's  software 
Isplsssntatlon  In  order  to  learn  ths  software  prior  to  officially 
accepting  software  support  responsibility. 

SPH  < IM ) -OOj ;  Oy^gT^QN :  The  dtsndsrds  for  softwsrs  Isplsssntatlon 
required  by  ths  procurement  activity  are  adequate. 


S£lilISll223l.9USSXIQ!iI  *1*^*  isplsssntatlon  methodology  used  by 
dsvslopssnt  contrsctor  Is  adsquste. 
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SPW<|M)"00<: _ QUgSIIQNl  Th«  i«pl«ii«nt«tlon  standards  and  nathoda 

adoptad^for  ~^uaa  by  tha  oparatlon  support  activity  during  post 
daployaant  aoftwara  support  ara  adaquata. 

SEB5IPjr99Zi--9^??TI9>^*  davalopaant  contractor  monitor  of 

subcontractor  aoftwara  inplaaantation  procaaa  has  baan  adaquata. 

3PH<IW>-00g;  question;  Tha  iaplaaantation  coaplation  of  CSCIa  has 
baan  raaaonabla  ralativa  to  tha  aoftwara  lifa  cycla  schadula. 

3PW<IW)-009;  gyKIIQW*.  Tha  procuraaant  aoftwara  projact  aanagamant 
support  tool  anvironaant  is  adaquata. 

Tha  davalqpaant  contractor  aoftwara  pro^act 
aanagamant  support  tool  anvironmant  is  adaquata. 

Tha  davalopaant  contractor  software 
configuration  managaaant  aupport  tool  anvironmant  is  adaquata. 

Tha  davalopaant  contractor  ayatam  aoftwara 
tool  anvironmant  is  adaquata. 

Tha  davalopaant  contractor  application 

aoftwara  taat  anvironmant  is  adaquata. 

SPWCIM)  *014:  Tha  oparatlon  aupport  aoftwara  aupport  tool 

anvironmant  is  adaquata. 

*23iI22l215i _ SUSSIISIfi  Tha  oparatlon  aupport  concapt  for 

implanantation  of  aoftwara  raviaiona  during  post  daployaant  software 
aupport  is  adaquata. 

The  operation  support  concapt  for 

intplamantation  audita  and  ravlaws  during  post  deployment  software 
aupport  la  adaquata. 


SOFTWARE  PROJECT  HANACEHENT  TEST  STRATEGIES 


SPW  <Taf)  :■  Tha  TEKP  adequately  dascribaa  tha  aoftwara 

taat  and  evaluation  procaaa. 

The  software  taat  procaaa  for  DT&E  has 

followed  tha  guidalinaa  in  tha  TEHP. 

S£3iXS2l923l..9y§SX£S!iI  The  software  test  procaaa  for  0T6E  has 

followed  tha  guidalinaa  in  tha  TEHP. 


S22!lJS2l995i _ 9y££T2222  The  laplamantation  of  tha  aoftwara  teat 

procaaa  by  0T6E  and  0T6E  organizations  has  baan  adaquata. 


-9y£STI9j? t«st  organlxatlon*  h«v«  Incorporated 
strategy  in  their  aoftwere  teat  proceaaea  for  coordination  an^{{r 
sharing  of  teat  plans,  procedures,  and  results. 

S£SSIS2zQ9§l~.^9SSSlIQlil  The  requiresenta  for  the  development 
contractor  aoftwere  test  strategy  are  clearly  specified  in  the  RFP, 
SOW,  snd/or  CORLs. 

gpg<Jg)“907;  OUgSJjtQNj  The  use  of  an  organization  for  software  test 
ZV6V  support  has  been  effective. 

SPff<X^)*,OOS;  The  overall  planning  for  software  tasting  has 

been  adequate. 

gPff The  software  teat  approach  and  methodologies 
esployed  are  clearly  described  in  the  software  test  docsentation  and 
appear  to  be  effective. 

??{?iIS2l9.t9i-9ySSTI9??i  The  software  features  to  be  tasted  and  not  to 
be  tested  are  clearly  described  in  the  aoftwere  teat  documentation. 


Spt^CTS) -Oil:  QUESTION:  The  traceability  of  software  features 
tasted/not  tasted  to  the  software  functional  raquiraments  ia 
described  in  the  software  test  documentation. 

zQlil  ■  9yf?TI9!*  t  The  software  test  deliverables  are  adequately 
specified  in  the  software  test  documentation. 

5?2iT5).;0i3i^QU|STJ2tii  The  software  teat  criteria  used  to  dateraine 
whether  each  teat  has  passed  or  failed  is  clearly  specified  in  the 
software  teat  documentation. 

§2i;iJS2l2i5i-9ys5Js2i!i  The  personnel  groups  responaibla  for  the 
software  teats  are  adequately  identified  in  the  software  test 
documentation . 

jTS) -0^3 ;  gUS^TION ;  The  high  risk  assumptions  of  the  software 
testing  approach  along  with  contingency  plana  for  each  such 
assumption  ia  adequately  described  in  the  software  test 
documentstion . 

553ir52z21§i-9y55TI21Ll  The  schedule  for  software  test  milestones  is 
adequately  specified  in  the  software  test  documentation. 

^?!1<T^) -0^7:  Software  testing  ia  adequately  prioritized  in 
the  software  test  approach  according  to  mission  criticality  concerns. 

SESlXSll21§1.9ySSXI&lil  The  software  test  environment  la  adequately 
identified  in  the  software  teat  documentation  and  is  adequate  for 
accomplishing  the  required  testing. 


5E3iX52l212i_9y£5JI22!2  The  conflguretlon  management  of 
-.atst  process  os  acequsts. 


the  aoftwe^^ 


t 
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SPM<Tg)-020;  QUgSTTQN;  Th«  tr«n«ltion  of  th«  ooftwaro  toat  atratagy 
fro»”tha”davaIop»ant"contract9r  to  tha  oparatlon  aupport  activity  haa 
baan  adaquataly  addraaaad  In  tha  aoftwara  taat  documantation  and  tha 
procuraaant  aoftwara  taat  plana. 


SOFTWARE  PROJECT  MANAGEMENT  PROJECT  INTERFACES 


S££i<£I>jSdli.Qy£SIIfil!i 

ara  adaquata. 

9£S<£I2rS02i.sy£2IIS£i 

ara  adaquata. 

S£I$<£I>jS23i.Qy£SII2Ni 

adaquata . 

5£i!i£I2jS2ii-SyS51Ifl‘fi 

adaquata . 

S£I{i£I2zog9i.Qyssiigi(i 
adaquata . 

S£fii£i2z2C$i.dy££TigNi 


Tha  ayataa  program  offica  axtarnal  Intarfacaa 
Tha  Implamanting  command  axtarnal  intarfacaa 
Tha  uaing  command  axtarnal  intarfacaa  ara 
Tha  aupporting  command  axtarnal  intarfacaa  ara 
Tha  training  command  axtarnal  intarfacaa  ara 
Tha  davalopmant  contractor  axtarnal  intarfacaa 


ara  adaquata. 

5£4?i£I2j222i-9y5SIJSUi  T*'*  O«vaiopmant  Taat  and  Evaluation  <D7&E> 
organization'axtarnal  intarfacaa  ara  adaquata. 

52Si£Iil22Si-Sli§5XIS?ii  Tha  Oparational  Taat  and  Evaluation  (OTfcE) 
organization  axtarnal  intarfacaa  ara  adaquata. 

5£Sl£I2j22§i-9i!£5I2Sli2  Tha  Computar  Raaourcaa  Working  Croup  (CRWG) 
axtarnal  intarfacaa  ara  adaquata. 

SPWi£Ili212l._9yHIISN^  Tha  Taat  Planning  Working  Group  (TPWG) 
axtarnal  intarfacaa  ara  adaquata. 


3>PM(I>J> -g^^j_9y£gI|gNj  Tha  Zntarfaca  Control  Working  Group  <ICWG) 
axtarnal  intarfacaa  ara  adaquata. 


SP^<?t).-91i?t,  Tha  Indapandant  Varification  and  Validation 
(IV&V)  agancy  axtarnal  intarfacaa  ara  adaquata. 

S£Si£J2j212i— 2y£5II9Jll!2  Tha  aoftwara  configuration  aanagamant 
intarfacaa  among  all  activitiaa'  managamant  componanta  for  tha 
aub^act  ayatam  ara  adaquata. 


S£l$i£I2j215i.2y£SJ221l!i  Tha  aoftwara  quality  aaauranca  managamant 
intarfacaa  among  all  activitiaa'  managamant  componanta  for  'cha 
aubract.  ivstam  sra  adacuatm. 


A2-9 


SP!1(PI)  -Q15:  _QUESTIONj.  Th«  contract  nanaganont  Intarfacaa  among 
aetlvltiaa'  managanant  conponanta  for  tha  aubjact  ayatam 
adaquata. 

5EiJi£I2r21§i-9i/S5II2ifi  Tha  intaraarvica  axtarnal  intarfacaa  with  ail 
activitlaa'  managaaant  componanta  ara  adaquata. 
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SOFTWARE  CONFIGURATION  HANAGEKENT  IDENTIFICATION 


IIB  L  It  -  -StfSSI  IQli  I  proeur«M«nt  policy*  stiandard*,  and 

convantiona  appliad  to  tha  idantlf ieation  of  aoftwara  configuration 

Itmmm  ara  adaquata. 

proouranant  idantif ication  of  dalivarabla 
aoftwara  configuration  itaaa  ia  adaquata. 


-003:  OU^^TIIQN:  Tha  procuraaant  activity  idantif  ication  of  tha 
aoftwara  configuration  baaalinaa  ia  adaquata. 


ayataa/aagaant  apacif ication  adequately 
idantif iaa  alaaanta  of  tha  aoftwara  functional  baaalina. 


parforaanca  raquiraaant  apacificationa 
adequately  idantif y  alaaanta  of  tha  aoftwara  allocated  baaalina. 

iaplaaantation  apacificationa  adaquataly 
identify  alaaanta  of  the  aoftwara  product  baaalina. 

2 l99Z I -SyiSXIQ^ L  identifier  charactariatica  for  aoftwara 

configuration  itaa  naaaa  ara  adaquata. 

S9i}iIS2l999i-9ySiSXlSlfi  davalopaant  contractor  internal  idantifiar 
naming  atandarda/convantiona  aatiafy  contractual  ragulationa. 


2SiSiI21l22ti— 3i2s§XIS?ii  Oavalopmant  contractor  identification 
atandarda  and  conventiona  can  be  tranaitionad  to  operation  aupport 
standarda  and  conventiona. 


JSSilSizSISi— — SlJSSXaSSi  Development  contractor  deliverable 
cenf iguration  itama  are  named  to  adaquataly  identify  multiple 
varaiona  and  variationa. 

SSM^iP2“212i_— 99£2XJ92!i  Development  contractor  identification 
procedurea  ara  atructured.  to  permit  eaay  addition*  deletion*  or 
modification  or  configured  itama  at  any  hierarchical  level. 

5S3H22-2i2i_— 9y52JJ51!i  Development  contractor  identification 
procedurea  for  addition*  deletion*  and  modification  of  configured 
itema  are  being  followed. 

5C?!_CJp2z2i3i_3yy9X2Si!2  phyaical  medium  of  configured  itama  ia 
adaquacaly  deacribad  by  tha  development  contractor  aofewara 
componant/itam  identification  achame. 

5SSiI22z215i _ 9955X12132  "^ha  development  contractor  aoftwara 

idantiflera  adequately  diatinguiah  among  different  atatea  (e.g.* 
aource*  object*  load*  core  imagea*  liatinga)  of  tha  aoftwara. 
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ch«ng 


SCM(ID>"015;  qUESTIOW;  Th«  dttv«lopii«nt 
control  forn  idontiflora  «r«  «d«quato. 


contractor 


aoftwara 


K2?iIfi2z21Si— — 9SI5I121!i  Subcontractor  configuration.  itam 
idantif lcation~*practicaa  ara  aonitorad  by  tha  davalopmant  contractor. 

SCWCID)  -017: _ SUSSIIQlii  Tha  docuaantatlon  which  collactlvaly 

idantlfiaa  tha  contant  of  a  configuration  itaa  ia  adaquata. 

3CW<|D)  ~01gil_9USSIIC^I  Softwar*  conflgurad  itaaa  which  iaplaaant 
aafaty  provlaiona  ara  adaquataly  idantifiad. 

^<^<Ig)20t?j.OV£STION;  Software  conflgurad  itama  which  iaplaaant 
coaputar/coaaunicationa  aacurlty  provlaiona  ara  adaquataly 
idantifiad. 

Tha  Idantlflcatlon  raquirananta  for  poat 
daployaant  aupport^ara^adaquataly  addraaaad  in  tha  CRLCHP  (CRISP,  0/S 
CMP}. 

5S3ilfi2-23l2— 99fSU91fi  Tha  autoaatad  aupport  too  la  for  poat 
daployaant  aupport  of  configuration  idantif ieation  ara  adaquataly 
addraaaad  in  tha  CRLCMP  (CRISP,  0/S  CMP> . 


SOFTWARE  CONFIGURATION  MANAGEMENT  CONTROL  ^ 

S£Si.££i2221i—9U55II2?ii  Tha  procuraaant  policy,  atandarda,  and 
convantiona  applied  to  tha  control  of  aoftwara  configuration  itama 

ara  adaquata. 

SCM(gC)  *00^ i _ QUEg7«SSf.:  Tha  procuraaant  activity  haa  iaplaaantad 

adaquata  aoftwara  conf iguration  aanagaaant,  baaad  upon  ragulationa. 
to  control  tha  functional  and  phyaical  character iatiea  of  all  CSCZa. 

5£3iSS2c223i— 92S5U21!i  The  procuraaant  configuration  aanagaaant 
planning  docuaanta  contain  aufficiant  guidance  for  configuration 
control . 

5S3iSS2c2252— 9215JJS2J2  The  davalopaant  contractor  configuration 
managamant  activitiaa  ara  adaquataly  aonitorad  by  tha  procuraaant 
activity. 


5£i-i2S22225i— 92S§IIS3i 

procaduraa  for  tha  Claaa 
catagoriaa)  ara  adaquata. 


Tha  procuramant 
I  and  Claaa  II 


configuration  control 
changaa  (or  equivalent 


§S3i2£2r22§i_9y5§TigNi  Th< 


uaa  of  daviationa  and  waivara  by  the 
davalopaant  contractor  which  could  affact  tha  auppoiT^^hility  of  tha 
aoftwara  haa  baan  adaquataly  controllad  by  tha  procuramant  activity/ 
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<CC)  -OOZl-StfS^IIQlil  T*'*  procurement  beeeline  control  forme  ere 
edequete . 

gCW (gg) ~00^:_QyggTJONj  The  procurement  conf Iguretion  control  boerd 
proceduree  ere  edequete. 

?g3iSS2r99Si,Sy£SIISl?.S  procurement  proceduree  for  turnover  end 
trenefer  of  conf iguretion  control  to  the  operetion  eupport  ectlvlty 
hee  been  edequetely  plenned. 

Development  contrector  conf  iguretion  control 
etenderde  end  conventione  een  be  treneitioned  to  operetion  eupport 
etenderde  end  conventione. 

5S2iSS2z211i— SySSIlSJfi  The  development  contrector  conf  iguretion 
control  boerd  hee  en  edequete  interfece  with  the  procurement  ectlvlty 
conf iguretion  control  boerd. 

552?i£S2z2132— The  development  contrector  conf  iguretion 
control  boerd  proceduree  ere  edequete  to  dietinguieh  between  herdwere 
end  eoftwere  feiluree. 

ggM(gg)-Q^^;  QUESTION;  The  development  contrector  conf iguretion 
control** proceduree  cen  be  treneitioned  to  or  ere  competible  with  the 
operetion  eupport  ectlvlty  plenned  conf iguretion  control  proceduree. 

S£t?iSS2lS25i.,9yS?TyPy *  The  development  contrector  eutometed*  eupport 
toole  for  conf iguretion  control  of  beeelinee  end  internel  development 
Ldentifieetione  ie  edequete. 

§S2iSS2r215i-SyS5JISlfi  The  development  contrector  eoftwere  change 
control  forme  ere  edequete. 

SCM^Cg2«22§i _ SySSIISlfi  Subcontrector  conf  iguretion  item  control 

precticee  ere  monitored  by  the  development  contrector. 

^CM  ( CC )_ *017 :  .QDESTigN {_  Configured  iteme  which  implement  eefety 
provieione  ere  edequetely  controlled. 

S£131££2c21§i.9y££J122f2  Softwere  configured  iteme  which  implement 
computer /communicetione  eecurity  provieione  ere  edequetely 
controlled. 

S£2?i£S2r92Si-9y£3IISy2  Oietrlbutlon  of  configured  item  chengee  from 
the  operetion  eupport  ectlvlty  to  the  field  ie  edequetely  controlled. 

SSSi£21l222i-9ys5XI2iSi  The  conf  iguretion  control  reeponeibility  for 
Integreting  computer  reeourcee  ’  into  the  eyetem  hee  remeined 
centralized  throughout  the  life  of  the  eyetem. 

<  CC)  r  _  Cy  The  conf  iguretion  control  requiremente  for 
poet  deployment  eupport  ere  edequetely  eddreeeed  in  the  CRLChP  (CHZS? 
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\ I  operation  support  coB£lguratlon 
boards  ara  adaquataly  daflnad  to  handla  aoftwara  changaa. 


contro^^ 


SC!^  < )  -0^3 1  OUgSTIOS :  Tha  autoaatad  support  teola  for  post 
daployaant  support  of  configuration  control  ara  adaquataly  addraaaad 
in  tha  CRLChP  (CRISP  and  0/S  C»P> . 


SOFTWARE  CONFIGURATION  MANAGEMENT  STATUS  ACCOUNTING 


SCH<SA)-001: _ OUSST|ION;  .TNa  proeuraaant  policy*  standards*  and 

convantiona  appliad  to  tha  configuration  status  accounting  of 
aoftwara  configuration  itaaa  ara  adaquata. 

SSUSS&lzSQ^l-^QJ^SSIIQlil  ‘Tha  proeuraaant  activity  has  iaplanantad 
adaquata  aoftwara  configuration  status  accounting*  baaad  upon 
ragulationa*  to  raport  tha  functional  and  physical  character iatica  of 
all  CSCIa. 

SSl?i2A2l9S3 1  -  n  proeuraaant  configuration  aanagaaant 

planning  docuaanta  contain  auffieiant  guidanea  for  configuration 
status  accounting. 

proeuraaant  activity  configuration  atatusn^ 
accounting  proeaduraa  ara  adaquata. 

5£SiSAil225i— SyS2II2*!li  davalopnant  contractor  intarnai 

configuration  status  accounting  proeaduraa  ara  adaquata. 

22Si2A  L*222L,9yS2TSS^I  Oavalopaant  contractor  configuration  stacua 
accounting  standards  and  convantiona  can  ba  tranaitionad  to  operation 
support  standards  and  convantiona. 

2S3i2&2lSS2l_-9VF2TI9y  ^  davalopsant  contractor  configuration 

status  accounting  has  an  adaquata  intarf aca  with  tha  '  procuramanc 
activity  configuration  status  accounting. 

5SSiS£2i992i..dy£SII92Ll  Tha  davalopnant  contractor  configuration 
status  accounting  proeaduraa  can  ba  tranaitionad  to  or  ara  conpatibla 
with  tha  operation  support  activity  planned  configuration  status 
accounting  proeaduraa. 

2SPi2^2r29§i,9y52TISllfI  davalopnant  contractor  autonatad  aupporr 
tools  for  configuration  atatua  accounting  of  baaalinaa  and  intarnai 
davalopnant  identifications  are  sdaqusta. 

*Sul5^2l2i2i—_ Sy£5229i!i  davalopnant  contractor  aoftwara 

configuration  status  accounting  forma  ara  adaquata. 

Subcontractor  configuration  item  configurating^ 

-itacua  ;GCCur.tir.3  ^rccacur^a  sra  rsonitorad  sy  -ha  :a'/isl.:-,-n«rto}v 
contractor. 


SC«<SA)-0a2;  oygSTIQN;.  Status  of  softwsro  configuration  Itama  which 
inplaaont  safoty~proviaiona  is  adaquatoly  raportad. 

SCW<SA>“0^3;  QUgSTIQML  Status  of  softwsra  configurad  itams  which 
implasant  computar/cosmunicstiona  sacurity  provisions  is  adaquataly 
raportad . 

i  ^61  _ gyiSTIQK  j  Tha  configuration  status  accounting 

raquirasanta'for  post  daploysant  support  ara  adaquataly  addrassad  in 
tha  CRLCHP  (CRISP  and  0/S  CNP> . 

gygglJgW-*  Tha  oparation  support  configuration  status 
aceotinting  procaduraa  ara  adaquataly  dafinad  to  handla  software 
change  reporting  raqulrasants. 

_ OyESIISllfi  The  autosstad  support  tools  for  post 

daploysant  support  of  configuration  status  accounting  are  adequately 
addrassad  in  tha  CRLCHP  (CRISP  and  0/S  CMP> . 


SOFTWARE  CONFIGURATION  MANAGEMENT  AUDIT  AND  REVIEW 


SCM(AR)  ■  .PySSIISSl  The  procurasant  policy*  standards*  and 
conventions  applied  to  tha  audit  and  review  of  software  configuration 

itass  ara  adequate. 

Sy^SIISIfi  The  procurasant  activity  has  isplasantad 
adequate  software  configuration  audit/raviaw  based  upon  regulations 
to  control  tha  functional  and  physical  characteristics  of  all  CSCZs. 

I  9y  gSIISifi  The  procurasant  configuration  managament 

planning  docusanta  contain  sufficient  guidance  for  configuration 
audit/raviaw. 

SS2i&82rS95i_9y£SIlSyi  Tha  conduct  of  forsal  raviawa  and  audits 
follows  a  forsst  based  on  tha  checklists  fros  HIL’‘STD-1321S* 
appropriately  tailored  for  tha  specific  software  audit/raviaw. 

The  software  product  acceptance  raquiranants 

ara  adequate. 

SSSSJiBlzSSSl _ SyfSIJyyi  The  davalopsant  contractor  internal 

configuration  audit/raviaw  process  facilitates  tha  development  of 
high  quality  production  software. 

-  ?^S3TI0N !  Configuration  audit/raviaw  interfaces  among 
procurement,  davalopnant  contractor*  and  oparation  support  activities 

ara  adequate. 

r  jr.zr  sccor  ronf  i  rurnti  jr. 
T.ar.agar.ant  ■^oci  support  facalitataa  tha  audit/ravtavi  of  tha  process 
Py  which  changes  are  incorporated  into  conf iguration  identifications. 
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gC?l< AR)  3Q09«  OUESTIOM:  Subcontractor  configuration  itaa  audlt/raviaw 
practlcaa  ara  aonltorad  by  tha  davalopaant  contractor. 

S(pW<AR)  “Q\Q!  Configured  Itaaa  which  iaplaaant  aafaty 
provlaiona  ara  adaquataly  audited  and  ravlawad. 

tdl  «  Software  configured  itaaa  which  iaplaaant 
coaputer/coaaunicationa  aacurity  provieiona  ara  adaquataly  auditad 
and  raviewed. 

S<P)1<AR>-01^;  „9^E3TTpWi  The  aoftware  conf igiuration  audit/reviaw 
requireaenta  for  poat  daployaent  aupport  are  adequately  addreaaed  in 
the  CRLCMP  (CRISP  and  0/S  CHP> . 

ggg</^R)2gl3i._0yggTTPH.;  Tha  automated  aupport  too  la  for  poat 
deployment  aupport  of  configuration  audit/raviaw  ara  adequately 
addreaaed  in  the 


THE  BOM  CORPORATION 


BON/ABQ-86-0090-TR 


ATTACHMENT  A3 

GLOSSARY  OF  TERMS  ' 

A3.1  introduction. 

a.  The  glossary  of  tarms  for  the  RAMSS  has  varied  as  the 
methodology  development  has  progressed.  Refer  to  B0M/A-84-322-TR 
(Final)  dated  September  28,  1984,  for  a  complete  glossary  of  terms 
relating  to  risk  assessment. 

b.  Some  terms  have  more  than  one  description;  when  this  Is  the 
case,  the  description  either: 

(1)  Are  significantly  different  between  sources  (though  the 
effective  meaning  may  be  not  much  different) 

(2)  Are  used  differently  (different  users  or  technical 
language) 

(3)  May  be  found  within  the  context  of  a  different  source 

(4)  Have  real  differences  In  meaning. 

Both  OoO  and  non-OoO  (e.g.,  FIPS  Pl’Bs,  N8S  Special  Publications) 
sources  are  used.  The  non-OoO  sources  and  terms  are  not  mandated  for 
our  use,  but  are  rather  Included  for  breadth  of  understanding,  for 
those  relevant  terms  conmonly  used  with  the  non-OoO  governmental 
and/or  private  sectors. 

c.  The  source  of  each  description  Is  Indicated  by  a  symbol  In 
parentheses  before  that  source's  term  description: 
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Tht  symbols 

(AFOTECPl) 

{AF0TECP3) 

(AFOTECPS) 

(AFRS5-43) 

(AFR800-14) 

(0O0480A) 

(ROWE) 

(CURRENT) 


TERMi 

(SYMBOLi.i) 
Otscriptloni.i... 
(SYMBOH.2) 
Description!. 2... 


(SYMBa.l.n) 

Descrlptloni.n.., 

TERM2 


TERMh 


used  and  corresponding  sources  are: 


AFOTECP .  800-2,  Volume  I.  10  Nov  82,  "Software  Test 
Manager's  Guide." 

AFOTECP  300-2,  Volume  III,  I  Jan  34,  "Software  Main¬ 
tainability  Evaluator's  Sulde." 

AFOTEC  800-2,  Volume  V,  25  Jul  83,  "Software  Support 
Facility  Evaluation— User's  Guide." 

Air  Force  Regulation  55-43,  "Management  of  Opera¬ 
tional  Test  and  Evaluation",  28  Jun  1985. 

Air  Force  Regulation  800-14,  Volume  I,  "Management  of 
Computer  Resources  In  Systems,"  12  Sep  75.. 

OoO  Standard  480A,  "Configuration  Control  -  Engi¬ 
neering  Changes,  Deviations  and  Waivers",  12  Apr  78. 

Rowe,  William,  An  Anatomy  of  Risk.  John  Wiley,  1977. 

Current  document  definition. 
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A3.2  GLOSSARY  OF  TERMS  FOR  DEVELOPING  AND  IMPLEMENTING  A  RISK 
ASSESSMENT  METHODOLOGY  FOR  SOFTWARE  SUPPORTABILITY. 

Allocattd  Baseline 

(0O0480A) 

See  Baseline. 

Allocated  Configuration  Identification 
(DO0480A) 

Current,  approved  performance  oriented  specifications  governing 
the  development  of  configuration  Items  that  are  part  of  a  higher 
level  Cl,  In  which  each  specification  (1)  defines  the  functional 
characteristics  that  arc  allocated  from  those  of  the  higher  level 
Cl,  (2)  establishes  the  tests  required  to  demonstrate  achievement 
of  Its  allocated  functional  characteristics,  (3)  delineates  neces¬ 
sary  Interface  requirements  with  other  associated  configuration 
Items,  and  (4)  establishes  design  constraints.  If  any,  such  as 
component  standardization,  use  of  Inventory  Items,  and  Integrated 
logistic  support  requirements. 

Application  Software 

(AF0TECP5) 

The  software  written  by  software  support  personnel,  or  purchased 
from  a  contractor,  used  directly  In  supporting  ECSs.  It  Is 
normally  used  for  simulation,  testing,  and  ECS  code  development. 

Automated  Software  Development  Tool 

(AF0TECP5) 

A  component  of  System  Software  that  assists  In  the  design,  Imple¬ 
mentation,  documentation,  and  verification  of  ECS  software. 

Availability 

(AFR800-14) 

A  measure  of  the  degree  to  which  an  Item  Is  In  the  operable  and 
comil table. state  at  the  start  of  the  mission,  when  the  mission  Is 
called  for  at  an  unknown  (random)  point  In  time.  (MIL-STO-721) 

(AF0TECP5) 

The  probability  that  a  system  Is  operating  satisfactorily  at  any 
point  In  time  when  used  under  stated  conditions. 

Available  Person  Time  (APT) 

(CURRENT) 

The  software  support  person-months  available  for  a  particular 
software  release  computed  as  the  product  of  the  release  duration 


THE  BOM  CORPORATION 


BOH/ABQ-86-0090-TR 


irt  months,  the  number  of  support  personnel,  and  the  percentage  of 
the  time  those  personnel  are  dedicated  to  the  subject  software 

release  (versus  shared  across  other  releases  or  other  software 

systems).  This  time  Includes  overhead  activity  directly  related 

to  the  subject  release.  The.  release  duration  Is  the  release 

engineering  completion  date  minus  the  release  start  date. 

Baseline 

(0O0480A) 

A  configuration  Identification  document  or  a  set  of  such  documents 
formally  designated  and  fixed  at  a  specific  time  during  a  Cl's 
life  cycle.  Baselines,  plus  approved  changes  from  those  base* 
lines,  constitute  the  current  configuration  Identification.  For 
configuration  management  there  are  three  baselines,  as  follows: 

a)  Functional  Baseline.  The  Initial  approved  functional  con- 
flguratlon  identificatl on . 

b)  Allocate  Baseline.  The  Initial  approved  allocated  con- 
flguration  Ident^fi cation . 

c)  Product  Baseline.  The  Initial  approved  or  conditionally 
approved  product  configuration  Identification. 

(ROWE) 

A  known  reference  used  as  a  guide  for  further  development 
activities. 

Baseline  Profile 

(CURRENT) 

See  Baseline  Software  Change  Profile. 

Baseline  Software  Change  Profile 
(CURRENT) 

The  set  of  numbers  (or  any  subset)  determined  by  specifying  the 
number  of  requests  per  release  for  each  request  category.  A 
request  category  Is  the  triple  (type,  priority,  complexity)  where 
type  Is  conversion,  enhancement,  or  correction;  priority  is 
emergency,  urgent,  or  normal;  and  complexity  Is  high,  medium,  low. 

Baseline  Software  Supportablllty  Estimate 

(CURRENT) 

See  User/Supporter  Baseline  Estimate 

Block  Release 

(CURRENT) 

See  Release. 
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Change  Control 
(0o04d0A) 

Sec  Configuration  Control 
Complexity  of  MA 
(CURRENT) 

See  Maintenance  Complexity 
Computer  Program 
(AFR800-14} 

A  series  of  Instructions  or  statements  In  e  form  acceptable  to  an 
electronic  con^uter,  designed  to  cause  the  computer  to  execute  an 
operation  or  operations. 

Computer  Program  Configuration  Item  (CPCI) 

(CURRENT) 

See  Computer  Software  Configuration  Item 
Computer  Resources 
(CURRENT) 

The  totality  of  computer  hardware,  computer  software,  personnel, 
documentation,  supplies,  and  services. 

(AFR800-14) 

The  totality  of  computer  equipment,  computer  programs,  associated 
documentation,  contractual  services,  personnel  and  supplies. 

Computer  Resources  Integrated  Support  Plan  (CRISP) 

(AFR55-33) 

The  CRISP  Identifies  organizational  relationships  and  responsi¬ 
bilities  for  the  management  and  technical  support  of  computer 
resources.  It  functions  during  the  full-scale  development  (FSO) 
phase  to  Identify  computer  resources  necessary  to  support  computer 
programs  after  program  management  responsibility  and  system  turn¬ 
over  are  transferred.  After  the  transfer,  the  CRISP  continues  to 
function  as  the  basic  agreement  between  the  supporting  and  using 
commands  for  management  and  support  of  Computer  resources. 

Computer  Resources  Working  Group  (CRW6) 

(CURRENT) 

A  group  comprised  of  all  the  participating  commands  (for  a 
particular  system)  which  writes  and  updates  the  Computer  Resources 
Integrated  Support  Plan  (CRISP).  The  group  Insures  that  necessary 
elements  of  the  CRISP  are  Included  1n  transfer  and  turnover 
agreements . 
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Computer  Software  Configuration  Item  (CSCI) 

(CURRENT) 

Set  Configuration  Item 
Configuration  Audit 
(CURRENT) 

The  process  of  verifying  that  a11  required  configuration  Items 
have  been  produced,  that  the  current  version  agrees  with  specified 
requirements,  that  the  technical  documentation  completely  and 
accurately  describes  the  configuration  Items,  and  that  all  change 
requests  have  been  resolved. 

Configuration  Control 

(0O0480A) 

The  systematic  evaluation,  coordination,  approval  or  disapproval, 
and  ieplementation  of  all  approved  changes  in  the  configuration  of 
a  configuration  Item  after  formal  establishment  of  Its  configura¬ 
tion  Identification. 

Configuration  Identification 

(0O0460A) 

The  current  approved  or  conditionally  approved  technical  docu¬ 
mentation  for  a  configuration  item  as  set  forth  in  specifications, 
drawings  and  associated  lists,  and  documents  referenced  therein. 

Configuration  Index 

(CURRENT) 

This  document,  produced  by  the  development  contractor,  reports  the 
current  status  of  configuration  Item  development  In  terms  of 
specifications  and  other  documents  that  depend  on  the  configura¬ 
tion,  such  as  qualification  Test  Plans  and  Procedures,  User 
Manuals,  and  the  Version  Description  Document.  It  lists  all  ECPs. 
and  SCNs  Incorporated,  approved  ECPs  not  yet  Incorporated,  and 
other  data. 

Configuration  Item  (Cl) 

(AFR800-14) 

An  aggregation  of  equipment/software,  or  any  of  Its  discrete 
portions,  which  satisfies  an  end  use  function  and  Is  designated  by 
the  Government  for  configuration  management.  CIs  may  vary  widely 
In  complexity,  size  and  type,  from  an  aircraft  or  electronic 
system  to  a  test  meter  or  .round  of  ammunition.  During  development 
and  Initial  production,  CIs  arc  only  those  specification  Items 
that  arc  referenced  directly  in  a  contract  (or  an  equivalent 
in-housc  agreement).  During  the  operation  and  maintenance  period, 
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any  repairable  item  designated  for  separate  procurement  Is  a 
configuration  Item  (AFR  65-3). 

Configuration  Management  (CM) 

-(0O0480A) 

A  discipline  applying  technical  and  administrative  direction  and 
surveillance  to  (1)  Identify  and  document  the  functional  and 
physical  characteristics  of  a  configuration  Item,  (2)  control 
changes  to  those  characteristics,  and  (3)  record  and  report  change 
processing  and  Implementation  status. 

Configuration  Management  Plan  (CMP) 

(CURRENT) 

A  document  which  describes  project  responsibilities  and  procedures 
for  Implementing  CM. 

Configuration  Management  System  (CMS) 

(AFOTECPS) 

A  system  applying  technical  and  administrative  direction  and 
surveillance  to  Identify  and  document  the  functional  and  physical 
characteristics  of  a  configuration  Item;  to  control  changes  to 
those  characteristics  and  to  record  and  report  change  processing 
and  Implementation  status. 

Configuration  Status  Accounting 

(OoOAdOA) 

the  recording  and  reporting  of  the  information  that  is  needed  lo 
manage  a  configuration  effectively,  including  a  listing  of  the 
approved  configuration  Identification,  the  status  of  proposed 
changes  to  the  configuration,  and  the  Implementation  status  of 
approved  changes. 

Consistency 

(CURRENT) 

A  measure  of  the  extent  the  software  products  correlate  and 
contain  uniform  notation,  terminology,  and  symbology. 

Conversion  (Adaptive)  MA 

(CURRENT) 

See  Maintenance  Type. 

Corrective  MA 
(CURRENT) 

See  Maintenance  Type. 
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Critical  Issues 
(AFOTECPl) 

Those  aspects  of  a  system's  capability,  either  -oeratlonal,  tech¬ 
nical,  or  other,  that  must  be  questioned  before  system's  overall 
worth  can  be  estimated  and  that  are  of  primary  Importance  to  the 
decision  authority  In  reaching  a  decision  to  allow  the  system  to 
advance  Into  the  next  acquisition  phase  (OoO  Directive  5000.3). 

Data  Item  Description 

(AFR800-I4} 

A  form  which  specifies  an  Item  of  data  required  to  be  furnished  by 
a  contractor.  This  form  specifically  defines  the  content,  prepa¬ 
ration  Instructions,  format  and  Intended  use  of  each  data  product. 

Descriptiveness 

(CURRENT) 

A  measure  of  the  extent  that  software  products  contain  Information 
regarding  Its  objectives,  assumptions.  Inputs,  processing,  out¬ 
puts,  components,  revision  status,  etc. 

Development  Contractor  Activity 

(CURRENT) 

Those  organizations  responsible  for  development  of  a  system  In 
order  to  achieve  an  Initial  operational  capability.  Organizations 
Include  the  prime  development  contractor  and  any  subcontractors  to 
the  prime  contractor. 

Documentation 

(AFOTECPS) 

All  of  the  written  work  describing  operating  and  maintenance 
procedures  for  a  system. 

Embedded  Computer  Resources 

(AFOTECPl) 

Computer  resources  incorporated  as  Integral  parts  of,  dedicated 
to,  required  for  direct  support  of,  or  for  the  uporadlng  or  modi¬ 
fication  .of  major  or  less  than  major  systam(s)  (excludes  AOP 
resources  as  defined  and  administered  under  AFR  300  series) 
(USAF/RO/LE  Policy  letter,  13  October  1981). 

Embedded  Computer  System  (ECS) 

(AFOTECPl) 

a)  A  computer  that  (s  Integral  to  an  electromechanical  system  and 
that  has  the  following  key  attributes: 
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(1)  Physically  Incorporated  Into  a  large  system  whose  primary 
function  Is  not  data  processing 

(2)  Integral  to,  or  supportive  of,  a  larger  systen  from  a 
design,  procurement,  and  operations  viewpoint 

(3)  Inputs  include  target  data,  environmental  data,  command 
and  control,  etc. 

(4)  Outputs  include  target  information,  flight  information, 
control  signals,  etc. 

b)  In  general,  an  embedded  computer  system  (ECS)  is  developed, 
acquired,  and  operated  under  decentralized  management  (OoO 
Directives  5000.1,  5000.2). 

Emergency  MA 

(CURRENT) 

See  Maintenance  Priority. 

Engineering  Chanr/e  Proposal  (ECP) 

(AFRS5-43) 

A  formal,  priced  document  (DO  Fomr  1692)  used  to  propose  changes 
to  the  contact  provisions  and  scope.  If  not  partially  waived  (see 
Contract  Change  Proposal),  and  to  the  configuration  item  baseline 
identification  especially  when  related  equipment,  critical  issues, 
Interfaces,  or  technical  manuals  are  affected  or  retrofit  Is 
Involved.  See  MIL-STOs  480,  481,  and  483;  and  AfR  400-3. 

Enhancement  (Perfective)  MA 

(CURRENT) 

See  Maintenance  Type. 

Estimated  Person  Months  Per  Change 
(CURRENT) 

See  Person  Months  Per  Change 
Estimated  Risk 
(CURRENT) 

See  Software  Support  ability  Risk 
Estimation 
(ROWE) 

The  assignment  of  probability  measures  to  a  postulated  future 
event. 
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Evaluated  Person  Months  Per  Change 
(CURRENT) 

See  Person  Months  Per  Change 
Evaluated  Risk 
(CURRENT) 

.  See  Software  Supportablllty  Risk. 

Evaluation 

(ROWE) 

Comparison  of  an  activity  performance  with  the  objectives  of  the 
activity  and  assignment  of  a  success  measure  to  that  performance. 

Evaluation  Criteria 

(AFOTECPl) 

Standards  by  which  achievement  of  required  operational  effective* 
ness/sultablllty  characteristics  or  resolution  of  technical  or 
operational  Issues  may  be  Judged.  For  full-scale  development  and 
beyond,  evaluation  criteria  must  Include  quantitative  goals  (the 
desired  value)  and  thresholds  (the  value  beyond  which  the  charac¬ 
teristic  Is  unsatisfactory)  whenever  possible  (OoO  Directive 
5000.3). 

Expandability 

(CURRENT) 

A  measure  of  the  extent  that  a  physical  change  to  Information, 
computational  functions,  data  storage,  or  execution  time  can  be 
easily  accomplished  once  the  nature  of  what  Is  to  be  changed  Is 
understood. 

(AF0TECP5) 

A  measure  of  the  ease  with  which  the  functional  capability  of 
computer  hardware  or  software  may  be  expanded. 

Facility 

(AF0TECP5) 

The  physical  plant  and  the  services  It  provides;  specific  examples 
are  physical  space,  electrical  power,  physical  and  electromagnetic 
(TEMPEST)  security,  environmental  control,  fire  safety  provisions, 
and  communications  availability. 
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Firnwar® 

(AFOTECPl) 

a)  Cooiputtr  programs  and  data  loaded  In  a  class  of  mcinory  that 
cannot  be  dynamically  modified  by  the  computer  during 
processing. 

b)  Hardware  that  contains  a  computer  program  and  data  that  cannot 
be  changed  In  Its  application  environment. 

Note  1.  Computer  programs  and  data  contained  In  firmware  are 
classified  as  software;  the  circuitry  containing  the  computer 
program  and  data  Is  classified  as  hardware  (Data  and  Analysis 
Center  for  Software). 

Functional  Baseline 

(0O0480A) 

See  Baseline. 

Functional  Configuration  Audit  (FCA) 

(DO0480A) 

The  formal  examination  of  functional  characteristics  test  data  for 
a  configuration  Item,  prior  to  acceptance,  to  verify  that  the  Item 
has  achieved  the  performance  specified  In  Its  functional  or 
allocated  configuration  Identification. 

Functional  Configuration  Identification 

(0O0460A) 

The  current  approved  technical  documentation  for  a  configuration 
Item  which  prescribes  (1)  all  necessary  functional  charscterls- 
t^cs,  (2)  the  tests  required  to  demonstrate  achievement  of 
specified  functional  characteristics,  (3)  the  necessary  Interface 
characteristics  with  associated  Cl’s,  (4)  the  Cl's  key  functional 
characteristics  -and  Its  key  lower  level  Cl's,  If  any,  and 
(5)  design  constraints,  such  as  envelope  dimensions,  component 
standardization,  use  of  Inventory  Items,  Integrated  logistics 
support  policies. 

High  Complexity  MA 

(CURRENT) 

See  Maintenance  Complexity, 

Historical  Maintenance  Profile 
(CURRENT) 

A  histogram  of  data  on  software  system  releases,  with  the  x-axIs 
representing  discrete  ranges  of  (available)  person-months  per 
change  and  the  y-axis  representing  the  number  of  software  system 
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rtitasts  that  fall  Into  each  x-axIs  discrete  range.  For  purposes 
of  analysis  or  Illustration,  the  axes  may  be  reversed. 

Independent  Verification  and  Validation  (IV&V) 

(AFOTECPl) 

An  Independent  assessment  process  structured  to  ensure  that 
computer  programs  fulfill  the  requirements  stated  In  system  and 
subsystem  specifications  and  satisfactorily  perform  the  functions 
required  to  meet  the  user's  and  supporter's  requirements.  IV&V 
consists  of  three  essential  elements:  Independent,  verification, 
and  validation: 

(1)  Independent.  An  organization/agency  Mhich  Is  separate  from 
the  software  development  activity  from  a  contractual  and 
organizational  standpoint. 

(2)  Verification.  The  evaluation  to  determine  whether  the 
products  of  each  step  of  the  computer  program  development 
process  fulfill  all  requirements  levied  by  the  previous 
step.. 

(3)  Validation.  The  Integration,  testing,  and/or  evaluation 
activities  carried  out  at  the  system/subsystam  level  to 
evaluate  the  developed  computer  program  against  the  system 
specifications  and  the  user's  and  supporter's  requirements 
(AFR  88-14). 

Initial  Operational  Capability  (IOC) 

(CURRENT) 

that  point  In  a  system's  life  cycle  when  the  agreed  upon  number  of 
production  systems  has  been  delivered  to  the  user  (using  command) 
for  operational  use. 

Instrumentation 

(CURRENT) 

A  measure  of  the  extent  that  software  products  contain  aids  that 
enhance  testing. 

Interface  Control  Working  Group  (ICWG) 

(MIL-STD-483) 

For  programs  which  encompass  a  system/HWCI/CSCI  design  cycle,  an 
ICWG  normally  Is  established  to  control  Interface  activity  between 
contractors  or  agencies.  Including  resolution  of  Interface 
problems  and  documentation  of  Interface  agreements. 
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Inttropcrablllty 

(AF0TECP5) 

A  mtasurt  of  tht  degree  to  which  computer  hardware/software  can 
Interface  to  and  operate  with  other  similar  computer  hardware/ 
software. 

Low  Complexity  MA 

(CURRENT) 

See  Maintenance  Complexity. 

Maintainability 

(AFOTECPS) 

The  probability  that  a  system  out  of  service  for  maintenance  can 
be  properly  repaired  and  returned  to  service  In  a  stated  elapsed, 
time. 

Maintenance  Complexity 
(CURRENT) 

The  general  degree  of  difficulty  to  complete  a  maintenance 
request:  high,  medium,  low. 

High:  An  MA  where  changes  are  In  requirements,  design,  code,  and 
test;  or  greater  than  10  percent  of  CSCI  Is  affected;  or  several 
modules  are  affected  by  the  change  (global  changes);  or  the  tech* 
nical  nature  of  the  change  requires  highly  specialized  personnel 
skills; .or  the  level  of  effort  .  by  personnel  Is  large. 

Medium:  An  MA  where  changes  are  In  design,  code  and  test;  or 

between  I  percent  and  10  percent  of  CSCI  Is  affected;  or  at  least 
two  modules  are  affected  by  the  change  (semi -local);  or  the  level 
of  effort  by  personnel  Is  average. 

Low:  An  MA  where  changes  are  isolated  to  qnly  one  unit  (e.g.,  one 
module/comp  11  at  ion  unit)  of  code;  or  no  more  than  1  percent  of 
CSCI  Is  affected;  or  the  level  of  effort  by  personnel  Is  minimal. 

Maintenance  Documentation 

(AFOTECPS) 

The  documentation  that  describes  the  maintenance  of  computer 
system  hardware  and  software. 

Maintenance  Priority 

(CURRENT) 

The  criticality  of  the  maintenance  request  In  order  to  preserve 
mission  readiness;  emergency,  urgent,  normal. 
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Effltrgtncy;  An  NA  requiring  all  available  personnel's  dedicated 
effort  to  correct  the  problem  as  soon  as  possible  (e.g., 
24  hours);  MIL-STO-1679  severity  code  I  or  2:  mission  termination 
or  severe  degradation. 

Urgent:  An  HA  requiring  next  "block  release”  turnaround; 

MIL-STO-1679  severity  code  3:  mission  Impact. 

Normal:  An  HA  not  In  the  Emergency  or  Urgent  categories; 

MIL-STD-1679  severity  code  4  or  5:  mission  Inconvenience. 

Maintenance  Profile 

(CURRENT) 

See  Historical  Maintenance  Profile. 

Maintenance  Request  Category 
(CURRENT) 

the  Identification  of  a  maintenance  request  by  specification  of 
the  maintenance  priority,  type,  and  complexity. 

Maintenance  Type 

(CURRENT) 

the  type  of  maintenance  actions  required  to  complete  a  maintenance  ' 
request:  conversion,  enhancement,  correction. 

Conversion  (Adaptive)  NA:  Any  change/effort  to  a  software  system 
'  which  Is  Initiated  as  a  result  of  changes  In  the  environment 
(e.g.,  hardware,  system  software)  In  which  the  software  systsm 
must  operate. 

Enhancement  (Perfective)  MA:  Any  change.  Insertion,  deletion, 
modification,  or  extension  iiade  to  a  software  system  to  meet  the 
evolving  needs  of  the  user. 

• 

Corrective  MA:  Any  change  which  Is  necessitated  by  actual  faults 
(Induced  or  residual)  In  a.  software  system. 

•Medium  Complexity  MA 

(CURRENT) 

See  Maintenance  Complexity. 

Modularity 

(CURRENT) 

A  measure  of  the  extent  that  a  logical  partitioning  of  software 
products  Into  parts,  components,  and/or  modules  has  occurred. 
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Normal  MA 
(CURRENT) 

Set  Maintenance  Priority. 

Operation  Support  Activity 
(CURRENT) 

Those  organizations  responsible  for  post  deployment  operation  and 
support  of  a  system.  Organizations  Include  the  using  conmand, 
supportlno  command,  contractors  (If  used),  and  test  and  evaluation 
agencies  (If  used). 

Operational  Effectiveness 

(AFOTECPl) 

The  overall  degree  of  mission  accomplishment  of  a  system  used  by 
representative  personnel  In  the  context  of  the  organization, 
doctrine,  tactics,  threat  (Including  cotMtermeasures  and  nuclear 
threats),  and  environment  In  the  planned  operational  employment  of 
the  system  (OoO  Directive  5000.3). 

Operational  Suitability 

(AFOTECPl) 

The  degree  to  which  a  system  can  be  satisfactorily  placed  In  field 
use,  with  consideration  being  given  to  availability,  compatibil¬ 
ity,  transportability.  Interoperability,  reliability,  wartime 
usage  rates,  maintainability,  safety,  human  factors,  manpower 
supportablllty,  logistic  supportablllty,  and  training  requirements 
(OoO  Directive  5000.3). 

Person-Months  per  Change  (PMPC) 

(CURRENT) 

Available  PMPC:  Raw  personnel  resources  workload  to  support  a 
user/supporter  baseline  workload  estimate  of  a  specified  number  of 
changes.  Computed  as  the  number  of  full-time  equivalent  personnel 
times  the  release  cycle  In  months  divided  by  the  total  nixnber  of 
changes . 

Estimated  PMPC:  An  estimate  of  a  personnel  resources  workload 
required  to  support  the  user/supporter  baseline  estimate.  This 
estimate  Is  computed  by  using  a  regression  equation  whose 
coefficients  are  derived  from  historical  maintenance  release  data. 

Evaluated  PMPC:  A  realistic  estimate  of  personnel  resources  work¬ 
load  effectiveness  to  support  the  user/supporter  baseline  estimate 
as  derived  from. an  evaluation  of  the  software  supportablllty  char¬ 
acteristics. 
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Personnel 

(CURRENT) 

See  Support  Personnel. 

Personnel  Skill  Level 
(CURRENT) 

A  subjective  Integer  rating  from  1  (lowest)  to  S  (highest)  of 
software  support  personnel  experience,  education,  and  specific 
task  responsibility  capabilities. 

Physical  Configuration  Audit  (PCA) 

(0O0480A) 

The  formal  examination  of  the  "as-built”  configuration  of  a  unit 
of  a  Cl  against  Its  technical  documentation  In  order  to  establish 
the  Cl's  Initial  product  configuration  Identification. 

Priority 

(CURRENT) 

See  Maintenance  Priority. 

Probability 

(ROUE) 

A  numerical  property  attached  to  an  activity  or  event  whereby  the 
likelihood  of  its  future  occurrence  is  expressed  or  clarified. 

Probability  Distribution 

(ROWE) 

The  representation  of  a  repeatable  stochastic  process  by  a 
function  satisfying  the  axioms  of  probability  theory. 

Probability  of  Occurrence 

(ROUE) 

The  probability  that  a  particular  event  will  occur,  or  will  occur 
In  a  given  Interval. 

Procurement  Activity 

(CURRENT) 

Those  government  organizations  responsible  for  assuring  delivery 
of  a  production  system.  Organizations  include  the  program  office. 
Implementing  command,  development  and  operational  test  and  evalua¬ 
tion  agencies,  and  appropriate  Independent  verification  and 
validation  agencies  If  used. 
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Product  Baseline 

(DO0480A) 

See  Baseline. 

Product  Configuration  Identification 
(0O0480A) 

The  current  approved  or  conditionally  approved  technical  documen¬ 
tation  which  defines  the  configuration  of  a  Cl  during  the 
production,  operation,  maintenance,  and  logistics  support  phases 
of  Its  life  cycle,  and  which  prescribes  (1)  a11  necessary  physical 
or  form,  fit  and  function  characteristics  of  a  Cl,  (2)  the 
selected  functional  characteristics  designated  for  production 
acceptance  testing,  and  (3)  the  production  acceptance  tests. 

Program  Management  Directive  (PMO) 

(AFR800-14) 

The  official  HQ  USAF  management  directive  used  to  provide 
direction  to  the  Implementing  and  participating  commands  and 
satisfy  documentation  requiraments.  It  will  be  used  during  the 
entire  acquisition  cycle  to  state  requirements  and  request  studies 
as  well  as  Initiate,  approve,  change,  transition,  modify  or  ter¬ 
minate  programs.  The  content  of  the  PMO,  Including  the  required 
HQ  USAF  review  and  approval  actions.  Is  tailored  to  the  needs  of 
each  Individual  program  (AFR  ^-2). 

Program  Management  Plan  (PMP) 

(AFR800-14) 

The  document  developed  and  Issued  by  the  Program  Manager  which 
shows  the  Integrated  time-phased  tasks  and  resources  required  to 
complete  the  task  specified  in  the  PMO.  The  PMP  is  tailored  to 
the  needs  of  each  individual  program  (AFR  800-2). 

Program  Management  Responsibility  Transfer  (PMRT) 

(AFR800-I4) 

That  point  In  time  when  the  designated  Supporting  Command  accepts 
program  management  responsibilities  from  the  Implementing  Command.  . 
This  includes  logistic  support  and  related  engineering  and  pro¬ 
curement  responsibilities  (AFR  800-4). 

Program  Support  Tools 

(AF0TECP3) 

General  debug  aids,  test/retcst  software,  trace  software/hardware 
features,  use  of  compller/link  editor,  library  management/con¬ 
figuration  management /text  editor/display  software  tools. 
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Program- Tfst  Plan 
(AF0TECP3) 

Stt  of  dtscriptlons  and  procedures  for  how  the  program  is  to  be 
(or  can  be.  or  has  been)  tested. 

Qual ity  Assurance  (QA) 

(CURRENT) 

All  actions  that  are  taken  to  assure  that  a  development  organiza¬ 
tion  delivers  products  that  meet  performance  requirements  and 
adhere  to  standards  and  procedures. 

Release 

(CURRENT) 

A  version  of  a  software  system  representing  either  the  Initial 
baseline  configuration  or  an  update  to  a  previous  version  that 
Incorporates  a  defined  set  of  software  change  requests.  Each 
release  becomes  a  new  baseline  configuration. 

Release  Engineering  Completion  Data 

(CURRENT) 

The  date  when  the  software  engineering  activity  for  a  release  Is 
complete.  The  software  engineering  activity  Includes  configura¬ 
tion  management,  quality  assurance,  and  software  maintenance 
project  phases  of  requirements,  design,  code,  unit  test,  Integra¬ 
tion  test,  and  operational  test.  .Activity  Including  "kit 
proofing,"  prom  burning,  and  In  general  ‘technical  order  modifica¬ 
tions  which  typically  occur  between  engineering  completion  and 
field  implementation  (distribution)  Is  not  included. 

Release  Field  Date 

(CURRENT) 

The  date  when  a  software  system  release  is  officially  distributed 
and  Implemented  In  the  field  for  operational  use. 

Release  10 

(CURRENT) 

A  unique  Identifier  for  a  software  system  release. 

Release  Start  Date 
(CURRENT) 

The  date  when  major  anal^ls  activity  related  to  a  specified 
release  begins  for  which  software  support  resources  are  required. 
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RtlUbilUy 

(ROUE) 

Tht  probability  that  tht  system  will  perform  its  required 
functions  under  given  conditions  for  a  specified  operating  time. 

Risk 

(ROUE) 

The  potential  for  realization  of  unwanted,  negative  consequences 
of  an  event. 


Risk  Acceptance 
(ROUE) 

Uillingness  of  an  individual,  group,  or  society  to  accept  a 
spetific  level  of  risk  to  obtain  some  gain  or  benefit. 


Risk  Acceptance  Function 
(ROUE) 

A  subjective  operator  relating  the  levels  of  probability  of 
occurrence  and  value  of  a  consequence  to  a  level  of  risk 
acceptance. 

Risk  Acceptance  Level 

(ROUE) 

the  acceptable  probability  of  occurrence  of  a  specific  consequence 
value  to  a  given  risk  agent. 

Risk  Acceptance  Utility  Function 


(ROUE) 

The  profile  of  the  acceptability  of  the  probability  of  occurrence 
for  all  consequences  involved  in  a  risk  situation  for  a  specific 
risk  agent. 

Risk  Agent 

(ROUE) 

A  person  or  group  of  persons  who  evaluates  directly  the 
consequences  of  a  risk  to  which  the  person  or  group  of  persons  is 
subjected. 

Risk  Assessment 

(ROUE). 

The  total  process  of  quantifying  a  risk  and  finding  an  acceptable 
level  of  that  risk  for  an  individual,  group,  or  society.  It 
Involves  both  risk  determination  and  risk  evaluation. 
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Risk  Assessment  Methodology  for  Software  SupportabHity  (RAMSS) 
(CURRENT) 

A  method  of  determining  the  disparity  between  the  estimated  risk 
(determined  from  the  support  concept,  baseline  software  support- 
ability  profile,  and  historical  maintenance  profile)  and  the 
evaluated  risk  (determined  from  a  conversion  of  the  software 
suppbrtablllty  evaluation  metrics). 

Risk  Consequence 

(ROWE) 

The  Impact  to  a  risk  agent  of  exposure  to  a  risky  event. 

Risk  Determination 
(ROWE) 

The  process  of  Identifying  and  estimating  the  magnitude  of  risk. 

Risk  Estimation 
(ROWE) 

The  process  of  quantification  of  the  probabilities  and  consequence 
values  for  an  Identified  risk. 

Risk  Evaluation  ^ 

(ROWE) 

The  complex  process  of  developing  acceptable  levels  of  risk  to 
Individuals  or  society. 

Risk  Profile  Baseline 

(CURRENT) 

The  measure  of  information  and/or  requirements  which  serve  as  the 
zero  reference  against  which  negative  (and  positive)  outcomes  can 
be  determined. 

Risk  Reduction 

(ROWE) 

The  action  of  lowering  the  probability  cf  occurrence  and/or  the. 
value  of  a  risk  consequence,  thereby  reducing  the  magnitude  of  the 
risk. 

Sensitivity  Analysis 
(ROWE) 

A  method  used  to  examine  the  operation  of  a  system  by  measuring 
the  deviation  of  Its  nominal  behavior  due  to  perturbations  In  the 
performance  of  Its  components  from  their  nominal  values. 
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Simplicity 

(CURRENT) 

A  measure  of  the  extent  that  software  products  reflect  the  use  of 
singularity  concepts  and  fundamental  structures  In  organization, 
language,  and  Implementation  techniques. 

Simulation 

(AFR800-I4) 

The  representation  of  physical  systems  or  phenomena  by  computers, 
models  or  other  equipment. 

Site 

(CURRENT) 

A  software  support  site,  or  particular  location,  where  software 
support  activity  Is  being  accomplished.  Includes  sites  such  as 
the  Air  Logistics  Centers  (ALCs). 

Site  Survey  Form 

(CURRENT) 

The  data  collection  form  used  during  the  software  support  site 
visits  to  collect  background,  evaluation,  and  maintenance  release 
data. 

Software 

(AFOTECPl)- 

A  set  of  computer  programs,  procedures,  and  associated  documenta¬ 
tion  concerned  with  the  operation  of  a  data  processing  system. 

(CURRENT) 

The  programs  which  execute  In  a  computer.  The  data  Input,  output, 
and  controls  upon  which  program  execution  depends  and  the  documen¬ 
tation  which  describes.  In  a  textual  medium,  development  and 
maintenance  of  the  program. 

Software  Change  Request 

(CURRENT) 

An  official  request  that  could  Involve  a  change  to  a  software 
system.  Such  requests  Include  problem  report,  enhancement 
requirement,  modification  request,  or  any  other  form  that  Is 
officially  tracked  by  a  configuration  management  function. 

Software  Configuration  Management 

(CURRENT) 

A  discipline  applying  technical  and  administrative  direction  and 
surveillance  co  i)  identify  and  document  :he  functional  and 


A3-21 


THE  BOM  CORPORATION 


MW/ABq-86-0090-TR 


physical  characteristics  of  a  configuration  ften,  2)  control 
changes  to  those  characteristics,  and  3)  record  and  report  change 
processing  and  Implementation  status. 

Software  Delivery 

(CURRENT) 

That  point  In  the  software  life  cycle  when  the  software  support 
function  assumes  responsibility  for  the  "next”  set  of  configura¬ 
tion  chanms  to  the  software  (e.g.,  next  block  rtlcase).  This 
point  Is  logically  no  later  than  PMRT,  but  could  bo  as  early  as 
IOC.  This  applies  when  a  contractor  or  government  agency  assumes 
the  software  support  function. 

Software  Error 

(CURRENT) 

The  human  decision  (Inadvertent  or  by  design)  which  results  In  the 
Inclusion  of  a  fault  In  a  software  product. 

Software  Fault 

(CURRENT) 

The  presence  or  absence  of  that  part  of  a  software  product  which 
can  result  In  software  failure. 

Software  Life  Cycle  Process 

(CURRENT) 

The  policy,  methodology,  procedures,  and  guidelines  applied  In  a 
software  environment  to  the  software  development  and  support  life 
cycle  activities. 

Software  Maintainability 

(AFOTECPl) 

The  ease  with  which  software  can  be  changed  in  order  to: 

(ij  Correct  errors 

(2)  Add/modify  system  capabilities  through  software  changes 

(3)  Delete  features  from  programs 

(4)  Modify  software  to  be  compatible  with  hardware  changes. 
(CURRENT) 

A  quality  of  software  which  reflects  the  effort  required  to  per¬ 
form  software  maintenance  actions. 
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Software  Maintenance 
(CURRENT) 

Thoet  actions  required  for: 

(1)  Correction  -  Removal,  correction  of  software  faults 

(2)  Enhancement  -  Additlon/deletlon  of  features  from  the 
software 

(3)  Conversion  -  Modification  of  the  software  because  of 
environment  (data  hardware)  changes. 

Software  Maintenance  Environment 

(CURRENT) 

An  Integration  of  personnel  support  systems  and  physical 
facilities  fur  the  purpose  of  maintaining  software  products. 

Software  Maintenance  Measures 

(CURRENT) 

Measures  of  software  maintainability  and  environment  capabilities 
to  support  software  maintenance  activity. 

Software  Maintenance  Project  Management 

(CURRENT) 

The  software  life  cycle  process  management  applied  during  the 
support  phrase  for  the  software  to  accomplish  specific  software 
maintenance  tasks  which  derive  from  software  problem  reports  or 
change  requests. 

Software  Management 

(CURRENT) 

The  policy,  methodology,  procedures,  and  guidelines  applied  in  a 
software  environment  to  the  software  development/maintenance 
activities.  Also,  those  personnel  with  software  management 
responsibilities. 

Software  Project  Management 

(CURRENT) 

See  Software  Management. 

Software  Project  Management  Design  Methods 
(CURRENT) 

The  software  project  management  process*  utilizes  design  methods 
which  enhance  software  sopportabllity  to  the  extent  that  design 
methodology  standards  and  conventions  are:  1)  documented, 

followed,  and  validated  through  quality  assurance,  2)  can  be 
transitioned  to  support  activity,  and  3)  produce  adequate  design 
specifications  which  reflect  supportaoility  characteristics. 
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Softwart  Project  Management  Implementation  Methods 
(CURRENT) 

The  software  project  management  process  utilizes  Implementation 
methods  which  enhance  software  supportablllty  to  the  exte- that 
Implementatlon/codlng/testlng  methodology,  standards,  and  conven¬ 
tions  are:  1)  documented,  followed,  and  validated  through  quality 
assurance,  2)  can  be  transitioned  to  the  support  activity,  and 
3)  produce  supportable  production  products. 

Software  Project  Management  Organization  Structure 

(CURRENT) 

The  software  project  management  process  organization  structure 
enhances  software  supportabITIty  to  the  extent  that  the  physical 
structure,  functional  responsibilities,  external  Interfaces  and 
assigned  personnel  provide  for  continuity  over  the  software  life 
cycle  phases,  and  have  proper  Interfaces  with  organizations 
responsible  for  software  support. 

Software  Project  Management  Planning 

(CURRENT) 

The  softwart  project  management  process  utilizes  planning  which 
enhances  software  supportablllty  to  the  extent  that  plans  for  the 
development,  test,  product  transfer,  operation  and  support  exist, 
have  been  Implemented,  have  been  appropriately  coordinated  across 
activities,  and  satisfy  contractual  and/or  regulation  require¬ 
ments  . 

Software  Project  Management  Project  Interfaces 
(CURRENT) 

The  software  project  management  possesses  organization  Interfaces 
which  enhance  software  supportablllty  to  the  extent  that  external 
project  organization  relationships  and  responsibilities  are: 
I)  defined,  2)  provide  a  valuable  functional  role,  and 
3)  contribute  to  systematic  cost  effective  procurement,  develop¬ 
ment,  operation  and  support  processes. 

Software  Project  Management  Test  Strategies 

(CURRENT) 

The  software  project  management  process  utilizes  test  strategies 
which  enhance  software  supportablllty  to  the  extent  that  the  test 
plans,  descriptions,  procedures,  and  results  have  been:  1)  docu¬ 
mented,  2)  can.  be  transitioned  to  the  support  activity,  and 
3)  provide  for  a  consistent  and  systematic  process  for  verifying 
and  validating  that  software  requirements  have  been  satisfied. 
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Software  Reliability 
(CURRENT) 

A  quality  of  software  which  refitcts  the  probability  of  failure 
frH  operation  of  a  software  component  or  system  In  a  specified 
environment  for  a  specified  Item. 

Software  Portability 

(CURRENT) 

A  quality  of  software  which  reflects  the  effort  required  to 
transfer  the  software  from  one  environment  (hardware  and  system 
software)  to  another. 

Software  Support  Concept 

(CURRENT) 

The  estimated  support  personnel  resources,  level  of  dedication  and 
expertise  of  the  support  personnel,  and  the  duration  of  the  block 
release  cycle. 

Software  Support  Facility  (SSF) 

(AF0TECP5) 

The  facility  which  houses  and  provides  services  for  the  support 
systems  and  personnel  required  to  maintain  the  software  for  a 
specific  ECS. 

Software  Support  Personnel 
(CURRENT) 

Sec  Support  Personnel. 

Software  Support  Resources 
(CURRENT) 

The  totality  of  personnel,  systems,  physical  facilities,  and 
calendar  time  that  are  used/consumed  during  a  software  support 
release  effort. 

Software  Supportabillty 

(CURRENT) 

A  measure  of  the  adequacy  of  personnel,  resources,  and  procedures 
to  facilitate: 

(1)  Modifying  and  Installing  software 
*  (2)  Establishing  an  operational  software  baseline 

3).  Meeting  user  requirements. 
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Software  Support ability  Evaluation 
(CURRENT) 

An  evaluation  to  derive  a  measure  of  how  well  a  software  system 
can  be  supported.  (See  Software  Supportablllty.) 

Software  Supportablllty  Evaluation  Metrics 

(CURRENT) 

The  closed-form  questionnaire  scores  for  each  software  support- 
ability  characteristic  In  a  software  supportablllty  evaluation  as 
well  as  the  values  computed  by  cumulating  lower  level  scores. 

Software  Supportablllty  Magnitude  of  Risk  Consequence 

(CURRENT) 

The  level  of  Impact  to  a  software  user  or  supporter  u  a  result  of 
the  risk  level  of  a  software  supportablllty  negative  outcome. 

Software  Supportablllty  Negative  Outcome 

(CURRENT) 

Any  outcome  for  which  the  software  support  resources  are  not 
adequate  to  accomplish  required  software  support. 

Software  Supportablllty  Risk 

(CURRENT) 

The  probability  at  a  given  point  during  the  software  support  phase 
that  the  software  maintenance  activity  specified  by  a  baseline 
software  supportablllty  profile  cannot  be  accomplished  with  the 
available  software  support  resources. 

Estimated  Software  Supportablllty  Risk:  An  estimate  of  the  soft¬ 
ware  supportablllty  risk  determined  by  the  area  under  a  normal 
distribution  curve.  The  area  Is  the  part  under  the  curve  greater 
than  the  subject  software's  available  person-months  per  change 
value  as  computed  from  the  software  support  concept  and  baseline 
software  change  profile.  The  normar distribution  curve  Is  deter¬ 
mined  by  using  the  estimated  person  months  per  change  as  the  mean 
and  the  standard  deviation  from  the  derivation  of  the  estimated 
person  months  per  change  regression  equation. 

Acceptable  Software  Supportablllty  Risk:  The  estimated  software 
supportablllty  risk  which  Is  agreed  upon  by  the  user  (using 
command)  and  supporter  (supporting  command)  as  a  result  of  the 
baseline  software  supportablllty  agreement. 

Evaluated  Software  Supportablllty  Risk:  An  approximation  to  the 
software  supportablllty  risk  computed  from  the  software  support- 
ability  evaluation  metrics.  The  computation  Is  derived  from  a 
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linetr  regression  model  using  the  software  life  cycle  process, 
software  product,  support  personnel,  support  systems,  and  support 
facility  as  the  five  regression  equation  factors. 

Measured  Software  Supportablllty  Risk:  Sec  Evaluated  Software 
Supportabillty  Risk. 

Software  System 

(CURRENT) 

A  set  of  software  (specifications,  programs,  and  data)  which 
constitutes  a  well-defined  major  function  or  group  of  functions. 

Typical  systems  Include  avionics  OFF,  ground  based  communications, 
missile  guidance,  simulation,  threat  generator,  ATE,  and  electro¬ 
nic  warfare. 

Software  System  Type 

(CURRENT) 

One  of  seven  classifications  of  a  software  system's  primary 
functional  mission:  ATO,  ATE,  C-E,  EW,  OFF,  SIM,  SUF. 

ATO:  Aircrew  Training  Device  or  Operational  Flight  Trainer  for 
training  and  support  of  an  operational  system,  usually  In  the  form 
of  a  fflockup  simulator. 

ATE:  Automatic  Test  Equipment  software  to  support  the  testing  of 
hardware  units  under  test  (UUT),  create  and  maintain  the  environ¬ 
ment  where  the  test  software  may  be  used,  or  prepare/analyze/main¬ 
tain  test  software. 

C-E:  Communications-Electronics  software  for  command  and  control, 
communications,  surveillance  and  warning,  air  traffic  control, 
intelligence,  and  other  related  functions. 

EW:  Electronic  Warfare  software  that  involves  the  use  of  electro¬ 
magnetic  energy  and  performs  functions  either  separate  or  integral 
to  a  larger  airborne  or  ground  system. 

OFF:  Operational  Flight  Frogram  sof  tware/firmware  that  is 

Integral  to  an  onboard  aircraft  computer  system  including  naviga¬ 
tion,  flight  control,  fire  control,  weapon  delivery,  electronic 
engine  control,  and  heads-up  display. 

SIM:  Simulation  Software  not  Included  as  part  of  the  ATD, 

including  simulation  models. 

SUF:  Support  Software  Including  application  support  software  and 
system  support  software  not  Included  in  any  other  category. 
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Specification  Change  Notice  (SCN) 

(CURRENT) 

The  SCN  Is  used  to  distribute  approved  page  changes  to  authorized 
users  of  baseline  documents  who.  In  turn,  are  responsible  for 
posting  the  updates. 

Source  Code 

(CURRENT) 

The  form  of  the  program  code  In  Its  source  language. 

Standards 

AF(JTECP3) 

Procedures,  rules,,  and  conventions  used  for  prescribing 
disciplined  program  design  and  Implementation. 

Support  Concept 

(CURRENT) 

The  software  support  concept  usually  specified  as  part  of  the 
CRISP  and  OS/CMP.  Also  Includes  that  part  of  the  support  concept 
necessary  to  establish  the  acceptable  risk  from  a  baseline  soft¬ 
ware  change  profile:  standard  release  duration,  nueber  of  support 
personnel,  average  skill  level,  percentage  of  personnel  dedicated 
to  releases,  support  facility,  etc. 

Support  Facility 

(CURRENT) 

The  physical  facility  resources  that  must  be  available  for  t.he 
software  support  resources  to  accomplish  a  specific  ta$k(s). 

Support  Personnel 

(CURRENT) 

A  general  term  for  personnel  (military,  OoO  civilian,  or  DoO  con¬ 
tractor)  whose  skills  are  necessary  to  directly  support  mission 
critical  system  software  maintenance.  Includes  but  is  not  limited 
to  management,  technical,  non-technical  support,  and  contractor 
personnel. 

Support  System 

(AF0TECP5) 

Any  automated  system  used  to  change,  test,  or  manage  the  con¬ 
figuration  of  ECS  software  and  associated  documentation.  Includes 
but  Is  not  limited  to  Host  Processor,  Software  Bench,  Laboratory- 
Integrated  Test  Facility,  Operational -Integrated  Test  Facility, 
and  Configuration  Management  System. 
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Support  System  Facility 
(AFOTECPS) 

The  facility  resources  that  must  be  available  for  the  software 
support  resources  to  accomplish  a  specific  taskCs). 

SystM  Software 

(AFOTECPS) 

All  of  the  software  that  Is  part  of  the  software  support  facility 
computer  system.  It  Is  never  or  seldom  accessed  directly  by  soft* 
ware  support  facility  personnel;  It  controls  the  processing  of 
application  software.  It  Includes  the  Operating  System,  Source 
Code  Editor,  Language  Translator,  Link  Editor /Loader,  Librarian/ 
File  Manager,  Data  Base  Manager,  and  Automated  Software  Develop* 
nent  Tool. 

Test  and  Evaluation  Master  Plan  (TEMP) 

(AFR55-43) 

An  overall  Test  and  Evaluation  (TIE)  plan  designed  to  Identify  and 
Intecrate  the  effort  and  schedules  of  all  TIE  to  be  done  in  an 
acquisition  program. 

Threshold 

(ROWE) 

A  discontinuous  change  of  state  of  a  parameter  as  Us  measure 
Increases.  One  condition  exists  below  the  discontinuity,  and  a 
different  one  above  It. 

Time  to  Complete  Maintenance  Request  (TC) 

(CURRENT) 

The  calendar  time  from  receipt  of  the  maintenance  request  by  the 
support  control  group  until  the  request  has  been  accepted  as  part 
of  an  operational  system  software  configured  release.  (This  does 
not  mean  the  configuration  Is  released  or  distributed,  and  this 
time  does  not  Include  this  additional  delay.  If  any.) 

Type 

(CURRENT) 

See  Maintenance  Type. 

Uncertainty 

(ROWE) 

The  absence  of  Information;  that  which  Is  unknown. 
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Urgent  MA 
(CURRENT) 

Stt  Maintenance  Priority. 

Verlficatlon/Valldatlon  (of  computer  programs) 

(AFR800-14) 

The  process  of  determining  that  the  computer  program  was  developed 
In  accordance  with  the  stated  specification  and  satisfactorily 
performs,  in  the  mission  environment,  the  functlon(s)  for  which  It 
was  designed. 


